Eric N. JohnsonLockheed Martin Assistant Professor of Avionics Integration
Daniel Guggenheim School of Aerospace EngineeringGeorgia Institute of Technology
Seminar at DLR, Braunschweig, GermanyMay 23, 2006
Georgia Tech UAV Research Facility:Navigation and Control Research Highlights
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 2
Outline
• Adaptive Control– GTMax research UAV– GTSpy small ducted fan– Edge aerobatic research aircraft
• Active Vision– Multi-University Research Initiative (MURI), Active-vision control
systems for complex adversarial 3-D environments– International Aerial Robotics Competition
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 3
Neural-Network Adaptive Control
ApproximateDynamicInversion
ApproximateDynamicInversion
Pseudo-Control
νPlantPlant
δ
Plant Inputs (Actual Controls)
+Reference Model
Reference Model
Command
rmν
PDControl
PDControl
NeuralNetworkNeural
Network
-
+ TrackingError
pdνadν−
Doesn’t require high fidelity nonlinear flight dynamics model
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 4
( ) azez −+=
11σ
( )xVW TTσyad ==νIn matrix form:In matrix form:
• Feedforward neural networks with a single hidden layer are universal approximators
Single Hidden Layer Neural Network
( )σ ⋅
( )σ ⋅
( )σ ⋅
( )σ ⋅
MM
x1
x2
xN1
M
y1
y2
yN3
V W
N2
N1 N3
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 5
Neural Network Adaptive Control
PDControl
PDControl
DynamicInversionDynamicInversion
NeuralNetworkNeural
Network
PlantPlantReference Model
Reference Model
-
++
TrackingError
Command
cmdδ ActuatorActuatorδν
rmν
pdνadν−
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 6
Neural Network Adaptive Control with Pseudo-Control Hedging
PDControl
PDControl
DynamicInversionDynamicInversion
NeuralNetworkNeural
Network
PlantPlantReference Model
Reference Model
-
++
TrackingError
CommandEstimateHedge
EstimateHedge
ActuatorActuatorcmdδ
hedgeν
δν
x
rmν
pdνadν−
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 7
Implications
• “Shelter” adaptive element from the adverse effects of plant input characteristics: – Linear dynamics, latency, actuator saturation, rate saturation, etc.
• Achievable adaptation performance is increased dramatically
• Adaptation is correct during saturation– Adaptive element can recover from a period of “faulty” adaptation
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 8
Georgia Tech R-Max: GTMax
• Yamaha R-Max, – 66kg – 3m Rotor
• Instrumented as a Research VTOL UAV
• More than 300 research test flights since March 2002
• Platform for DARPA Software Enabled Control program rotary wing final experiments and demonstrations in Summer 2004
GTMax Automatic landing during November 2002 SEC Meeting
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 9
GTMax Avionics HardwareManual pilot
antenna
Datalink antennae
Vibration isolated avionics rack
GPS antenna
Sonar
• Flight Computers– 266MHz & 800 MHz
embedded PCs, Ethernet, flash drives
• Sensors– Inertial Measurement Units
(x2)– Differential GPS– Magnetometer, sonar
altimeter– RPM, voltage/fuel warnings– Various camera systems
• Data Links– 11 Mbps Ethernet data link– RS-232 serial data link
• Flight Computers– 266MHz & 800 MHz
embedded PCs, Ethernet, flash drives
• Sensors– Inertial Measurement Units
(x2)– Differential GPS– Magnetometer, sonar
altimeter– RPM, voltage/fuel warnings– Various camera systems
• Data Links– 11 Mbps Ethernet data link– RS-232 serial data link
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 10
Ground Control StationLaptopsInside
LaptopsInside
Datalink 2Datalink 2
Datalink 1Datalink 1
Ref. GPSRef. GPS
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 11
GTMax Avionics Installation and Wiring
Primary Flight Computer
Serial Extension Board
FreewaveDGR-115
DC/DC
5V
5V
Battery 12V
Aux
iliar
yM
odul
e
AironetMC4800
EthernetHub
NovAtel RT-2 GPS Receiver
Secondary Computer(Image Processor)
FCS20 Small Autopilot
ISIS-IMU
12V
DC/DC
DC/DC
HMR-2300Magnetometer
DC/DC
Sonar Altimeter
Power DistributionModule
Generator
Yamaha AttitudeControl System
RC Receiver YACS IMU
12V5V
RS-232 SerialEthernetDC Power
Dat
a Li
nkM
odul
eG
PS M
odul
eFC
S20
GPS
Flig
ht C
ompu
ter
Mod
ule
5V
12V
Pan/tilt/roll hardware(power wiring not shown)
Video Transmitter
Camera(several types)
Servos
EncodersEncoders
Encoders
ServosServos
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 12
GTMax Onboard Baseline Software
• Navigation– 17 State Extended Kalman
Filter Navigation System• Position, velocity, attitude• Accelerometer/gyro biases• Terrain height error
– All Attitude Capable– 100 Hz updates
• Control– Adaptive Neural Network
Trajectory Following Controller – Neural Network: 7 Outputs for
7 Degrees of Freedom– Full speed envelope– High performance position
tracking
• Datalink and Comms– Handles all serial devices – Reroute data over Ethernet,
save to file, playback– Software reconfigurable
• Trajectory Generation– Generates smooth
position/velocity/accel/attitude commands as a function of time from waypoints
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 13
Neural Network Adaptive Flight Control
• Simultaneous adaptation in 7 degrees of freedom• Address large model errors and disturbances
– GTMax model is based on a linear model for hover - adaptation used on for flight over the entire flight envelope of the helicopter
– Different configurations, damage
• Accounts for actuator saturation and “two-loop” design
• Tested over full speedenvelope, aggressive maneuvers
InnerLoop
External Command
OuterLoop
Online trainedNeural
Network
Attitude corrections
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 14
Automatic Takeoff
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 15
20 ft. lateral step response
Automatic Rapid Reposition
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 16
Automatic Landing
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 17
Auto Pirouette
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 18
Auto 180o Velocity Change
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 19
Automatic Flight at 95 ft/sec
9160 9180 9200 92200
20
40
60
80
Time (sec)
Gro
und
Spe
ed (f
t/sec
)
Upwind Downwind
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 20
Many Configurations…
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 21
Rotor synchronizes with video at 900RPM
Using RPM to Accommodate Frozen Collective Actuator (10 ft Climb)
10ft
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 22
Flying 50 ft square
Frozen “Right” Swash-Plate Actuator
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 23
Small Autopilot Development (FCS20)
• 60mm x 90mm x 32mm, 120 grams– DSP for floating point operations (1.3Gflops)– FPGA IO interface– MEMS rate gyros, accelerometers– Air data– GPS
• Same software as baseline GTMax– 17 state extended Kalman filter navigation– Neural network adaptive trajectory-following flight control– Trajectory generator
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 24
GTSpy Description
• Modified from MASS Helispy• FCS20 Small autopilot• Weight - 5 lbs• Height - 27 in• Duct Diameter - 11 in• Hover Endurance - 25 min
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 25
Autopilot Tests on GTSpy, Takeoff
5 pounds11 inch duct
(movie) http://uav.ae.gatech.edu/videos/s040807k1_takeoff.mpg
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 26
Autopilot Tests on GTSpy
5 pounds11 inch duct
(movie) http://uav.ae.gatech.edu/videos/s040807k3_right40ft.mpg
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 27
Autopilot Tests on GTSpy, Landing
5 pounds11 inch duct
(movie) http://uav.ae.gatech.edu/videos/s040807k5_landing.mpg
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 28
First Air Launching of a Hovering AircraftNovember 20, 2004
• GTMax at 400 feet• GTSpy Launched, then
hovers 80 feet below• Both vehicles utilizing the
same NN adaptive controller software
(movie 1) http://uav.ae.gatech.edu/videos/fs041120c1_launchHeliSpyLong.mpg (movie 2) http://uav.ae.gatech.edu/videos/fs041120c1ob_launchHeliSpy.mpg
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 29
Agile Maneuvering Test Aircraft: GTEdge
• 33% scale Edge 540T
• Avionics– FCS20 Small Autopilot– NovAtel DGPS– FreeWave wireless serial
• Flights began June 2005
60 ft/sec Autoflight(movie) http://uav.ae.gatech.edu/videos/e050624b3_60fps.mpg
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 30
Automatic Transition to Zero Speed• First transition to near-vertical attitude
and near-zero airspeed, July 2005– Video begins at about 880 sec– Slow level-flight envelope
expansion from forward flight to hover
0
500
1000
-500
0
500
10000
500
1000
1500
North (ft)
trajectory
West (ft)
heig
ht (f
t)
measuredcommandend
800 850 900 9500
20
40
60
time (sec)
spee
d (ft
/sec
)
800 850 900 9500
20
40
60
80
time (sec)
pitc
h an
gle
(deg
)
800 850 900 950-100
-50
0
50
100
time (sec)
elev
ator
(% o
f max
)
800 850 900 9500
20
40
60
80
100
time (sec)
thro
ttle
(%)
800 850 900 950-100
-50
0
50
100
time (sec)
aile
ron
(% o
f max
)
800 850 900 950-100
-50
0
50
100
time (sec)
rudd
er (
% o
f max
)
(movie) http://uav.ae.gatech.edu/videos/e050723c3_vertical.mpgand e050723c2_30fps20fps.mpg
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 31
High Speed: 160 ft/sec (95 knots)
(movie) http://uav.ae.gatech.edu/videos/e060128a2_160fps.mpg
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 32
Rapid Transition to Hover & Return to Forward Flight
(movie) http://uav.ae.gatech.edu/videos/e060509a1_rapidTransition.mpg
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 33
Other Collaborations
AFRL & Aerovironment, Skytote
DARPA & Frontier Systems / Boeing, Maverick/Renegade
D6
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 34
Active-Vision Control Systems for Complex Adversarial 3-D Environments• AFOSR/AFRL MURI: Georgia Tech, MIT, UCLA,
Virginia Tech• Development of sound methods that utilize 2-D and
3-D imagery to– Enable aerial vehicles to autonomously detect and prosecute targets in
uncertain complex 3-D adversarial environments– Do these things without relying upon highly accurate 3-D models of the
environment– Include capabilities and approaches inspired by those found in nature
• New strategies of – Target recognition/tracking– Obstacle/hazard avoidance– Navigation, guidance, and control
Active-Vision Control
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 35
Tracking with Region-Based Deformotion
• Simultaneous segmentation, registration flow
• Registration using energy minimization with gradient descent
• Handle occlusions
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 36
Vision-Only Guidance, Navigation, and Control
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 37
The Measurements
Available measurements from a candidate window• Size• Rotation• Horizontal pos. in camera• Vertical pos. in camera
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 38
Flight Test Results
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 39
Vision-Aided Inertial Navigation
• Inertial Navigation System (INS) aided by 2-D vision sensor looking at a selected image target
Autopilot UAV
IMU + Navigation
Camera Image Processing+Computer Vision
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 40
Vision-Aided Inertial Navigation
• Approach– Use assumed range (altitude), position in image, size of object in
image, and aircraft state to estimate object position, size, andorientation
– Use subsequent measurement of object image position and size in image to update INS
– Utilize inertial data to maintain lock on target
– Operate without GPS (or any other position aiding)
Last Object LocationLast Object Location
Actual NewObjectLocation
ExtrapolatedNew Object Location∆z
∆y
Actual NewObjectLocation
ExtrapolatedNew Object Location∆z
∆y
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 41
Vision-Aided INS Trials
Special Purpose Optical Target
Using Existing Features(in this case a window)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 42
2GT
Close Approach to Building, no GPS
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 43
Flight Test Results: Error Plots
error = |estimate – GPS|
Max Errors:9.6 ft (x)8.3 ft (y)4.0 ft (z)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 44
Air-to-Air Vision-Based Tracking
Guidance Autopilot UAV
IMU + Navigation
Camera
Estimated:LOS RateRangeObject Size
Acc Com
PositionVelocityAccelerationRates
Kalman Filter Image Processing+Computer Vision
No information is communicated between aircraft, and only passive 2-D vision information is available to maintain formation
Camera
Y
X
y
x
Camera Image Plane
Target
Image frame
Z
u = (uX, uY, uZ) : unit vector α : subtended angle
r : range b : target size
Estimation Stateu, u, , , b. 1 r
r r .Estimation State
u, u, , , b. 1 r r r
.u, u, , , b. 1 r
r r u, u, , , b. 1 r r r
.
Measurementu, α
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 45
• Fast marching methods + target acquisition process= 10 frames/sec
Video 1 Video 2
Tests on Recorded Videos
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 46
Estimator DesignApply Extended Kalman Filter (EKF) to• Measurement
calculated by using image processor outputs
• Estimation state
Unit vector expressed in a camera frame
Unit vector expressed in a NED frame
Leader
Camera frame
Xc
Yc
Zc
North
EastDown
Follower
u : unit vector
α : subtended angle
r : range
leader’ssize : b
alat : leader’s lateral acceleration
Measurement and Estimates
Leader’s lateral accelerationIn the leader’s wind frame( perpendicular to leader’s velocity vector )
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 47
6-DOF Image-in-the-Loop Sim Results
• Relative position • Relative velocity
ImageProcessor EKF Controllerz x̂
Simulator
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 48
Simulation Results: Image Processor• Image source : recorded video in flight test• Simulated camera motion : circling with a constant speed• Commanded camera position : 100ft behind, 20ft below the leader
• Image processor outputs
Leader’s positions in image Unit vector and subtended angle
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 49
Simulation Results: 3D EstimatorPosition
Velocity Acceleration
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 50
Flight Test Results• Image processor outputs and EKF estimates
Unit vector and subtended angle
0.94 0.96 0.98 1 1.02 1.04 1.06
x 104
-0.5
0
0.5
1
navTime
u
u1u2u3
0.94 0.96 0.98 1 1.02 1.04 1.06
104
0.05
0.1
0.15
0.2
0.25
navTime
α
Leader position
0.94 0.96 0.98 1 1.02 1.04 1.06
x 104
0
1000
2000
navTime
X
VISIONGPS
0.94 0.96 0.98 1 1.02 1.04 1.06
x 104
0
500
1000
1500
navTimeY
VISIONGPS
0.94 0.96 0.98 1 1.02 1.04 1.06
104
-500
-400
-300
-200
navTime
Z
VISIONGPS
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 51
Formation Flight Tests (GTMax/GTEdge)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 52
Vision-Based Formation Flight Tests (GTMax/GTEdge)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 53
Status
• Currently “closing the loop” on guidance –true vision-based formation flight
• First attempts were earlier this week• Longest run 22 seconds• Current efforts include more advanced vision
processing, estimation, and guidance• Moving into other intended scenarios:
see/avoid, pursuit
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 54
Box Target Maneuver Sinusoidal Target Maneuver
Neural Network augmenting
estimator corrects bias
}}
Estimating Dynamics of Maneuvering Target
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 55
International Aerial Robotics Competition
Launch area
Two lights near building
Vehicle or subvehicle(s) enter building
Building and an Entry Point Found
>1m
Transmit an image of “point of interest”inside building
In less than 15 minutes:
Image receiver(& other ground components)Sign on
building
3 km
http://avdil.gtri.gatech.edu/AUVS/CurrentIARC/2001CollegiateRules.html
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 56
Levels
Level 1: Follow prescribed waypoints for 3kmLevel 2: Locate building and find an entry
• Level 3: Operate inside the building – Can be a different vehicle or subvehicle than used above– Can launch near target structure
• Level 4: All of the above– Complete in < 15 minutes
Contest gets new rules once somebody does level 4
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 57
AUVSI InternationalAerial Robotics Competition
Launch AreaIdentify building and map entry point usingonboard systems
>1m
Image receiver(For judges)
Sign on correctbuilding
2003 ApproachGoal: Level 2 Mission
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 58
Simulation and Flight Tests in Preparation for 2003 Competition
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 59
2003 Results, Level 2
• Allowed Four Mission Attempts• On Three of the Attempts:
– Building Located by Recognizing Symbol• 1m x 1m sign• Fifteen buildings to chose from
– Valid Opening Found and Automatically Reported to Judges– Took 16-17 minutes each time
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 60
Future Plans
• Continue to expand on air-to-air tracking– Cluttered background– Flight testing and facilitate technology transition
• Air-to-ground tracking (by airplanes)– Leverage air-to-air work (particularly cluttered background and
guidance work)
• Obstacle avoidance– Initial emphasis on unknown fixed obstacles, ownship state known– Using passive monocular 2-D sensor
• Requires considerable work in image processing, 3-D estimation, and guidance
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 61
Expected Outcomes and Transitions
• New capabilities of autonomous sensing and control, enabling operations: – In the presence of uncertainty– In a clandestine/covert manner– In close proximity to hazards, structures, and/or terrain
• Relevant flight test validation• Enable more capable/reliable existing air vehicles
and guided munitions• Enable entirely new systems to be developed
(for example, capable of operating in urban environments)
June 06 ENJ - Georgia Tech - uav.ae.gatech.edu 62
Acknowledgements
• Sponsors: DARPA, AFRL, AFOSR, NSF, NASA, Lockheed Martin• Partners: GST, Boeing, Draper, Lockheed Martin, Northrop Grumman,
Honeywell, McKenna MOUT, SSCI, OGI, Virginia Tech, MIT, UCLA• Georgia Tech Participants:
Thomas Apker, Anthony Arkwright, Mike Baldwin, Sean Barnes, Anthony Calise, Cesar Carrillo, Jane Case, Claus Christmann, Henrik Christophersen, Scott Clements, Mike Curry, Joerg Dittrich, Graham Drozeski, Jason File, Nate Fisher, Lucas Fortier, Stewart Geyer, Richard Giuly, Luis Gutierrez, Bryan Gwin, Jincheol Ha, Greg Ivey, Mat Hart, Bonnie Heck, Jeong Hur, Anna Ivester, Ole Jakobsen, Eric Johnson, Phillip Jones, Suresh Kannan, Adrian Koller, Dan Laszweski, Tom Linden, Sumit Mishra, Alex Moodie, Kyungjin Moon, Hiong Chair Naik, Wayne Pickell, Seung-Min Oh, J.V.R. Prasad, Alison Proctor, Nimrod Rooz, Bhaskar Saha, Daniel Schrage, Liang Tang, Allen Tannenbaum, Mike Turbe, Shannon Twigg, Suraj Unnikrishnan, George Vachtsevanos, Patricio Vela, Yoko Watanabe, Linda Wills, Allen Wu, Ilkay Yavrucuk, and many others…