Sense and Avoid (SAA) Sense and Avoid (SAA) T h l i f U dT h l i f U dTechnologies for Unmanned Technologies for Unmanned

Aircraft (UA)Aircraft (UA)Aircraft (UA)Aircraft (UA)

National Cheng Kung University4 December 2008ece be 008

Dr. Won-Zon Chen

1

Outline

• Sense and Avoid (SAA) background and needSense and Avoid (SAA) background and need• SAA technology overview

– SAA architectureSAA architecture– Sensor suite– Sensor data fusion– Autonomous collision avoidance

• SAA flight test• Hardware-in-the-loop (HWIL) lab

2

Presentation ReferencesPresentation References

• “A Real-time Electro-Optical (EO) Sensor System Simulator for• A Real-time Electro-Optical (EO) Sensor System Simulator for Unmanned Vehicle SAA Development,” AUVSI Conference, San Diego, CA, June 2008

• “Sense and Avoid Flight Test (SAAFT) and Lessons Learned,” AIAA InfoTech Conference, Rohnert Park, CA, May 2007

• “SAA and TCAS Integration for Unmanned Aerial System (UAS),” AIAA S a d C S teg at o o U a ed e a Syste (U S),InfoTech Conference, Rohnert Park, CA, May 2007

• “SeFAR Integration Test Bed for SAA Technologies,” AIAA InfoTech Conference Washington D C September 2005Conference, Washington, D.C., September 2005

• SAA video, AFRL 2006

3

SAA Background and Needg

Unmanned Aircraft (UA) must meet or exceed “equivalent level of safet ” (ELOS) to ha e ro tine access to the NAS and be able tosafety” (ELOS) to have routine access to the NAS and be able to integrate with existing processes and procedures These are four layers involved in achieving air traffic separation

Procedural

Air TrafficManagementAir Traffic

ManagementConflictAvoidance

Onboard Collision Avoidance System

Pilot See and AvoidCollisionAvoidance

Sense and Avoid for UAV

4

SAA Challengesg

Must meet or exceed equivalent level of safety (ELOS)M t b li ht i ht ll i l d ff d blMust be light weight, small size, low power, and affordableMust comply/compatible with existing infrastructureMust meet UAV unique ConOps and flying characteristics q p y g

Low maneuverability and speedData link latency and reliabilityGround pilot situation awareness

Cooperative

Non-Cooperative

SAAFT Objective: In-flight demonstration of autonomous detect and

5

SAAFT Objective: In flight demonstration of autonomous detect and autonomous avoid for cooperative and non-cooperative intruder aircraft

Maturing SAA in StepsNGC SAA Integration Lab

• TCAS + ADS-B + EO + Radar Simulators• Real-time Closed-loop Simulation up to 10

Intruder Aircraft

SAA Flight Test (SAAFT)Surrogate Programmed to Fly Like Global• Surrogate Programmed to Fly Like Global Hawk

• Autonomous Avoidance with TCAS + EO• 3 Rounds of Flight Test from Oct 2006 to

Jan 2007Jan 2007Multiple Intruder Autonomous Avoidance (MIAA)

• Autonomous Avoidance with EO + Radar + TCAS + ADS-B

• 2008 - 2009

P d iProduction Transition

6 Algorithm and Software Development

AFRL SAA VideoAFRL SAA Video

0603111 See & Avoid (REVIEW) mpg0603111 See & Avoid (REVIEW).mpg

7

Outline

• Sense and Avoid (SAA) background and need• Sense and Avoid (SAA) background and need• SAA technology overview

– SAA architectureSAA architecture– Sensor suite– Sensor data fusion– Autonomous collision avoidance

• SAA flight test• Hardware-in-the-loop (HWIL) lab

8

SAA Architecture

Sensors Data Fusion AutonomousAvoidance

9

Traffic Collision Alert System (TCAS)y ( )

• Developed in late 1980s after mid-air collision at Los Angeles

• TCAS provides two primary functions– Surveillance

• Interrogates transponder equipped vehicles• Filters and tracks transponder responses to provide vehicle position and

velocity information– Traffic advisories

• Uses track data to estimate threats Traffic alert (TA)• Recommends pilot action Resolution alert (RA)

• Pilot Has Ultimate Responsibility To Respond To ThreatsThreats– TCAS I: TA only– TCAS II: TA and RA

10

TCAS II Block DiagramgDirectional Antenna

Radar Altitude

PressureAltitude

TCAS PMode-S

Transponder

(top)Altitude Altitude

TCAS Processor Transponder

TCAS

RA Display

Control Panel

RA

Omni DirectionalAntenna

Aural Annunciation

Traffic DisplayDisplay(bottom)Traffic Display

11

TCAS II Operational Descriptionp p

TCAS Goal to OptimizeVertical Separation atVertical Separation at

Closest Point of Approach

Time to Closest Point of ApproachTime to Closest Point of Approach~ 40 Seconds (Traffic Advisory)

Time to Closest Point of Approach~ 25 Seconds (Resolution Advisory)

12

Automatic Dependent Surveillance -Broadcast (ADS-B)Broadcast (ADS B)

• Broadcast ownship position to provide air traffic controller-like situational awareness

• Include vehicle position, altitude, and intent in data message• Long range and improved accuracy • Rebroadcast via ground station• Include ground Radar information in rebroadcast data message

• Provide other possible applicationsProvide other possible applications• Weather information• Terrain information

C tl th iGarmin’s UAT

• Currently three versions • Universal Access Transceiver (978 MHz)• Mode-S (1090 MHz)

VDL VHF– VDL – VHF• Asia and Europe Only

13

ADS-B Ground Station

ADS-B

ADS-B, TIS-B,

ADS-B

ADS B, TIS B, FIS-B

FISFIS Sources

UATTx/Rx

GroundStationsFIS

ADS-BSurveillance

Systems

FIS-BGround BroadcastServer(s)

ADS B

TIS-BSurveillance ProcessingSystems

Radars

ADS-B

Flight Following

TargetReports

Processing

14

Control Facility (WJHTC) (Limited)TargetReports

ADS-B/TIS-B : Principle of Operations

Class A Airspacep

Transponder N T d

PrimaryRadar

TransponderEquippedAircraft

Non-TransponderEquippedAircraft

RebroadcastEquipment

RebroadcastEquipment

TransponderEquippedNon-Transponder

E i dTransponder

EquippedAircraft

q ppAircraftEquipped

AircraftPrimaryRadar

15100nmi

EO Sensor System

• Automatic traffic detection and tracking via advanced image processing

16

EO Calibration

• Significant optics Radial Distortion

0 8

distortion due to wide camera FOVs

• The intrinsic camera 0.4

0.5

0.6

0.7

0.8

r [D

eg]

Camera1Camera2The intrinsic camera

calibration process computes

Focal Length-0.1

0

0.1

0.2

0.3

0 10 20 30 40 50

Erro

r

Camera3

– Focal Length– Principal Point – Skew Coefficient

R di l & T ti l

Azimuth [Deg]

– Radial & Tangential Distortion

• Line-of-Sight angles t d f i lcorrected from pixel

measurements via distortion map

17

ICX’s AI-130 RadarICX s AI 130 Radar

Performance• Frequency: ~ 35 GHz• Frequency: ~ 35 GHz• Detection Range: ~ 6 nmi• Azimuth Coverage: ± 85 deg• Elevation Coverage: ± 10 degElevation Coverage: ± 10 deg• Scan Rate: 120 deg/sec• Range Resolution: < 5 metersPhysicaly• Antenna Diameter: 11”• Weight: 55 lb• Power: 250

W 28 dW, 28 vdc• Environmental: -40 to +70 C• Altitude: up to 20 kftCostCost• ~ $ 170K

18A new SAA radar is being developed

Sensor Data IntegrationSensor Data IntegrationSensor Data IntegrationSensor Data Integration

Data IntegrationgFault

Detection & Isolation

Track Mgmt

Sensor Data

Fused Tracks& Isolation

Data Assoc

Data Fusion

LAPEOi

Track-EO

fusion

Track files

Track

LAPEOi

Track-EO

fusion

Track files

TrackGlobal Tracks

LAPTCASj

Track-TCAS

Track files

LAPTrack

to Track Fusion

Global Tracks

LAPTCASj

Track-TCAS

Track files

LAPTrack

to Track Fusion

19

j

fusionj

fusion

Collision Avoidance: Concept of Operation

Intruder

p p

LOSRate

Jointly OptimalPaRCA

Trajectory---

Intruder4-D Tube

AutoACASTrajectories

Trajectory

+++

---

LOS

4 D Tube

ATC Corridor ++

+++

Vector

Passive RangingAcceleration

+

Bootstrapmaneuver

20Ownship

Constrained Optimization for Collision Avoidance

Stepwise Constraint Relaxation

Avoidance

“SAA system selects appropriate response for situation”

p

ATC • ATC clearances can be recoordinated

Mi i t i t t tManeuver limits (comm, sensor, etc.)

• Mission constraints are next to go

• RoW rules obeyed until last FARs & Right of Way rules

yinstant

• Ultimately, only the limits of the airframe are observed in orderManeuver limits (airframe) airframe are observed in order to save the aircraft

21

Achieving “Equivalent Level of Behavior” to Manned A/C

Outline

• Sense and Avoid (SAA) background and need( ) g• SAA technology overview

– SAA architecture– Sensor suite– Sensor data fusion

A t lli i id– Autonomous collision avoidance

• SAA flight testH d i th l (HWIL) l b• Hardware-in-the-loop (HWIL) lab

22

SAAFT Flight Test in December 2006 g

• SAA host Learjet configuration– Electro optical (EO) and – Traffic Collision Avoidance System (TCAS) sensors

TCAS Display

RLGINSTCAS

TCASAntenna

VSS

RLG

CCDCCD

INS

CCD2CCD2

Detector

Detector

TrackingProcessing

PaRCAHost

VSS(GH-Like

Response)

SAA

CCD1CCD1

DetectorCCD3CCD3

SAAMonitor

DAAMonitor

GPS

23

5NP45-009

Surrogate UAS – Learjet In-Flight Simulator

• Owned/operated by Calspan• FAA registered as experimentalFAA registered as experimental• Four degree-of-freedom in-flight

simulation– Pitch, roll, yaw, thrust

Programmed to fly like HALE

EO Radome

• Programmed to fly like HALE

EO Radome

IR Radome

24

MIAA Flight Test in 2008/2009g

TCAS DisplayTCAS Display

TCAS

p y

TCASAntenna

RLGINSTCAS

p y

TCASAntenna

RLGINS

Detector

TrackingPaRCA

Host

VSS(GH-Like

Response)

CCD1CCD1 Detector

TrackingPaRCA

Host

VSS(GH-Like

Response)

CCD1CCD1

CCD2CCD2Detector

Detector

gProcessing

CD3CD3

SAAMonitor

DAAMonitor

GPS

CCD2CCD2Detector

Detector

gProcessing

CD3CD3

SAAMonitor

DAAMonitor

GPS

5NP45-009

CCD3CCD3

5NP45-009

CCD3CCD3

RadarRadarRADAR

ADS-BA tADS-BA t ADS-B

AntennaAntennaRADAR

25

AntennaAntenna ADS B

MIAA Round 1 Successfully Completed in MIAA Round 1 Successfully Completed in May 2008May 2008May 2008 May 2008

One week long flight test for EO and Radar performance characterization: 20+ test points in various encounter geometries and atmospheric20 test points in various encounter geometries and atmospheric conditions

Three EO Cameras

EO and RadarCameras

MISTY

100 Mi

MISTY MOA

30 M

i Restricted

(Approximation) Finished InstallationMISTY

MOA Test Site

Installation on

Calspan Learjet

26

FAA Supportpp

• FAA provided the SAAFT program: TSPI & ADS-BTSPI & ADSTSPI & ADS--BBp p g– Two intruder vehicles & flight crews

• King Air – representing “general aviation” intruder (small)C “

Z-XtremeSelf-Contained Unit

L1/L2

GPS S67-1575Antenna

ADS-B• Convair – representing “transport” intruder (large)

– Test equipment and engineering support• Automatic Dependent Surveillance-Broadcast

ADS BTransceiver

GDL 90Data Computer

Automatic Dependent Surveillance Broadcast (ADS-B)

• Time Space-Position Information (TSPI)ADS-B UAT 978 Mhz

Top / Bottom Antenna’s UAT

27Beech King Air N35 – “general

aviation” size intruderConvair 580 N39 – “transport”

size intruder

Representative Single-Intruder Test Geometries

• Level overtake • Ascending head-onAscending head on

• Level head-on

• Descending head-on

• Level abeam

28

Representative Multiple-Intruder Test Geometries

• Head-on, lateral separation • Crossing intrudersHead on, lateral separation g

• Head-on vertical separation• Head-on, vertical separation

29

Flight Test Results: Closed-loop Collision AvoidanceAvoidance

“…This is How Pilot Would Have Done It...” Stated Dana Purifoy, Calspan Chief Test PilotCalspan Chief Test Pilot

LJ and Convair approach each other head on with 300 feet vertical and 0.2 nm lateral offsetsTA issued at 5.1nmEO detect and PaRCA

Play Movie

EO detect and PaRCA maneuver (bank to the right as required by right-of-way rules) initiated at 4 5s405_final_overlay_xvid.avi 4.5nmRA issued at 3nm and PaRCA follows the RA command to descend andcommand to descend and then level offLJ passes Convair well clear and scenario

30

clear and scenario terminated

Outline

• Sense and Avoid (SAA) background and need( ) g• SAA technology overview

– SAA architecture– Sensor suite– Sensor data fusion

A t lli i id– Autonomous collision avoidance

• SAA flight testH d i th l (HWIL) l b• Hardware-in-the-loop (HWIL) lab

31

SAA HWIL Lab

C FunctionalScene Generation

& IVMC’s Functional Equivalent

VMC’sTCAS II

Simulation Rack

Visual Displays

& Image Processing

Rack

VMCVMC Development

StationVehicle

Simulators & Control Station

32

Control Station

SAA HWIL Lab Details

33

TCAS HWIL Simulator

34

Real-time EO Sensor System Simulatory

• Vehicle SimulatorVehicle

Simulator

– Simulates Ownship and up to 10 intruder aircraft

IEEE 1394A

• Scene Generators– Simulate the

Ownship’s on-b d EO

Scene Generator

Scene Generator

Scene Generator

board EO sensors• Image Processors

– Process images d f

Ethernet TCP

Ethernet TCP

Ethernet TCP

and extract feature points of interest

• EO Track Manager

ImageProcessor

ImageProcessor

ImageProcessor

Ethernet UDP

– Associate related feature points into tracks of possible intruder aircraft

EO TrackManager

35

intruder aircraftIEEE 1394B

Scene Generation

• X-Plane version 8.60COTS Flight Simulator– COTS Flight Simulator

– Aircraft 3D Models– High Quality Graphics– Accurate Optical Effectsp

• Sun Glare, Shadows, Reflections

• Weather Management• Camera Position & FOV

-30° 30°

• X-Plane Plug-in SDK– C programs written by 3rd

party to work with X-Plane corecore

– Offers programmers access to control X-Plane’s internal variables

– Used to drive graphics of simulated vehicles

– Allow access to X-Plane OpenGL graphics

36

p g p

HWIL Lab Validation

HWIL Lab Flight 2936

37

Summaryy

• SAA capability comparable to manned aircraft is required for UA to fly freely in national and international airspace

• Northrop Grumman team has been developing enabling SAA technologies under Air Force Research Laboratory (AFRL)technologies under Air Force Research Laboratory (AFRL) sponsorship – Algorithm and software– System integration– System integration– Flight demonstration

• Northrop Grumman’s SAA HWIL lab has been used to facilitate SAA t h l d l tSAA technology development– Virtual prototyping– Lab validation

S t tifi ti t ti– System certification testing

38