64
MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson, Jonathan Las Fargeas, Jinwoo Seok Department of Aerospace Engineering University of Michigan Ann Arbor, Michigan September 2012 ARCLAB (UM) Collaborative Unmanned Air Vehicles September 2012 1 / 60

MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

  • Upload
    others

  • View
    5

  • Download
    0

Embed Size (px)

Citation preview

Page 1: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

MAX: Collaborative Unmanned Air VehiclesRecent Progress at UM

Anouck Girard & Pierre KabambaBaro Hyun, Justin Jackson, Jonathan Las Fargeas, Jinwoo Seok

Department of Aerospace EngineeringUniversity of MichiganAnn Arbor, Michigan

September 2012

ARCLAB (UM) Collaborative Unmanned Air Vehicles September 2012 1 / 60

Page 2: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Introduction

ARCLAB in Numbers

Current People:

3 PhD

2 MS

Publications:

10 peer reviewedjournal articlesaccepted

41 conferencepapers accepted

2 book chapterspublished

Graduated Students:

4 PhD:Justin Jackson, 2012, Llamasoft.Baro Hyun, 2011, Hyundai Motors.Christopher Orlowski, 2011, US Army, TACOM/TARDEC.Andrew Klesh, 2009, JPL.

6 MS:Zahid Hasan, 2012, Raytheon Company.Calvin Park, 2012, North American Bancard.Clarence Hanson, 2011, Rockwell Collins.Jonathan White, 2008, US Coast Guard.John Baker, 2007, Systems Engineering, HDT Robotics.

Amir Matlock, 2007, JHU Applied Physics Lab, Ballistic

Missile Defense Test and Evaluation Group.

ARCLAB (UM) Collaborative Unmanned Air Vehicles September 2012 2 / 60

Page 3: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

New Members of the ARCLab

Moritz Niendorf

Work Experience and Education

02/2011 - 07/2012: DLR (German AerospaceCenter) - Department for Unmanned Aircraft

11/2010: Diploma in Aerospace Engineering -University of Stuttgart, Germany

09/2008 - 05/2009: Exchange Student -Aeronautical and Astronautical Engineering -Purdue University

Research Interests

Mission and path planning for unmannedaircraft under motion constraints.

Task assignment for unmanned aircraftconsidering path planning aspects.

ARCLAB (UM) Collaborative Unmanned Air Vehicles September 2012 3 / 60

Page 4: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

New Members of the ARCLab

Dave Oyler

Work Experience and Education

Texas A&M University

05/2012: B.S. Electrical Engineering

NASA Johnson Space Center

05/2012-08/2012: Robotic Operations

01/2011-08/2011: Robotic Systems Technology

01/2010-05/2010: Integrated Communications

06/2008-08/2008: Electromagnetic Systems

Research Interests

Robotic planetary exploration

Cooperation of heterogeneous robotic teams

ARCLAB (UM) Collaborative Unmanned Air Vehicles September 2012 4 / 60

Page 5: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Overview

Mixed-Initiative Nested Classification

Themes

From ... Sensor, To ... Information

Trusted highly-autonomous decision-making systems

Objectives

Improve the classification performance in mixed-initiative system

ARCLAB (UM) Collaborative Unmanned Air Vehicles September 2012 5 / 60

Page 6: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Overview

Persistent Visitation, Detection, and Capture

Themes

Coherent change detection for persistent surveillance systems

Increased operational efficiency and autonomy

Objectives/Results

Formulation of the Persistent Visitation problem for a single UAV.

Proof of the existence of periodic paths for single UAVs performingpersistent visitation.

Complete algorithm to find minimal cost paths when fuel constraintsare considered.

Formulation of the Persistent Visitation, Detection, and Captureproblem for multiple UAVs.

Algorithm to generate paths for UAVs that perform persistentvisitation while attempting to image intruders.

Potential ImpactsARCLAB (UM) Collaborative Unmanned Air Vehicles September 2012 6 / 60

Page 7: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Overview

Mixed-Initiative Nested Classificationfor n Team Members

Baro Hyun, Songya Pan, Pierre Kabamba, Anouck Girard

Department of Aerospace EngineeringUniversity of Michigan, Ann Arbor, MI

Annual MACCCS Review

September 2012

B. Hyun et al. (UM) Inverting the ratio September 2012 7 / 60

Page 8: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Mixed Initiative Nested Classification

Motivated by military operations

Intelligence, Surveillance, and Reconnaissance missions

Objects of interests

threat or friend

Unmanned aerial vehicles(UAVs)

carry a suite of sensors and acommunication device

Human operators

direct the UAVsinspect data and makeclassification decisions

Need high quality classificationdecisions in the presence ofuncertainties

B. Hyun et al. (UM) Inverting the ratio September 2012 8 / 60

Page 9: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Mixed Initiative Nested Classification

Motivated by military operations

Intelligence, Surveillance, and Reconnaissance missions

Objects of interests

threat or friend

Unmanned aerial vehicles(UAVs)

carry a suite of sensors and acommunication device

Human operators

direct the UAVsinspect data and makeclassification decisions

Need high quality classificationdecisions in the presence ofuncertainties

B. Hyun et al. (UM) Inverting the ratio September 2012 8 / 60

Page 10: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Mixed Initiative Nested Classification

Information overflow

Multiple views from a wide angle camera (Gorgon Stare)

Figure: from Cummings and Bertuccelli, MAX review 10’

“... man power requirements to deal with these data are burdensome”[AF/ST, Report on Tech Horizon ’10]

B. Hyun et al. (UM) Inverting the ratio September 2012 9 / 60

Page 11: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Mixed Initiative Nested Classification

Objectives of the researchGlobal Objective

Improving the classification performance in mixed-initiative systems

Year 2-3 (2008-2010)

A mobile classifier taking multiple measurements while seekingmaximum information

Discrete Event System (DES) modeling of human operator inclassification task

A mobile classifier making sequential decisions while seekingminimum risk

Year 4-5 (2010-2012)

Information-classification performance, classification mechanism bythresholding, team classification

Inverting the human-to-machine ratio

B. Hyun et al. (UM) Inverting the ratio September 2012 10 / 60

Page 12: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Mixed Initiative Nested Classification

Motivational questions

How do we leverage the complementary strengths of human/machinecollaboration in a mixed-initiative system?

How can we invert the current human-to-machine ratio in the ISRmission?

Mixed-initiative system

1 Classifiers with workload-independent performance (machines)

2 Classifiers with workload-dependent performance (humans)

- “First-order” models to capture the features of machines andhumans, respectively

B. Hyun et al. (UM) Inverting the ratio September 2012 11 / 60

Page 13: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Mixed Initiative Nested Classification

Motivational questions

How do we leverage the complementary strengths of human/machinecollaboration in a mixed-initiative system?

How can we invert the current human-to-machine ratio in the ISRmission?

Mixed-initiative system

1 Classifiers with workload-independent performance (machines)

2 Classifiers with workload-dependent performance (humans)

- “First-order” models to capture the features of machines andhumans, respectively

B. Hyun et al. (UM) Inverting the ratio September 2012 11 / 60

Page 14: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Mixed Initiative Nested Classification

Technical relevance to the A.F.

Air Force relevance (Highlights from [Tech. Horizon])

From ... Sensor, To ... Information

“The volume of sensor data from current-generation sensors ... hasbecome overwhelming, as manpower requirements to deal with thesedata have placed enormous burden on the Air Force.”“... systems that can reliably make wide-ranging autonomous decisionsat cyber speeds to allow reactions in time-critical roles far exceedingwhat humans can possibly achieve.”

Grand Challenges for Air Force S&TChallenge #2: Trusted highly-autonomous decision-making systems“... demonstrate technologies that enable current human-intensivefunctions to be replaced, in whole or in part, by more highlyautonomous decision-making systems, ...”

B. Hyun et al. (UM) Inverting the ratio September 2012 12 / 60

Page 15: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Mixed Initiative Nested Classification

Literature survey

ClassificationTheory of classification [Gupta and Leu ’89, Widrow ’63]Applications of classification [Jain et al. ’00, Chang et al. ’06]Classification with human inputs [Cebron and Berthold ’06, Holsappleet al. ’08]

Statistical decision makingHypothesis testing [Lehmann and Romano ’10]Bayesian decision theory [Berger ’85]Sequential Probability Ratio Test (SPRT) [Wald ’45]

Human-machine collaborationInverting the ratio [Cummings et al. ’08-’10]Adjustable autonomy [Goodrich et al. ’09-’10]

B. Hyun et al. (UM) Inverting the ratio September 2012 13 / 60

Page 16: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Mixed Initiative Nested Classification

Technical contributions

We extended our work on mixed-initiative nested thresholding, aclassification architecture that uses a primary workload-independentclassifier and a secondary workload-dependent classifier, for a generalnumber n of classifiers in the architecture, formally pose the problem,and solve it.

We identified the optimal ratio of mixed-initiative team members, thecorresponding minimal probability of misclassification, and theindividual workload applied to the workload-dependent classifier as afunction of the total workload applied to the architecture.

We performed a sensitivity analysis of the aforementioned results withrespect to the peak performance of the workload-dependent classifier.

B. Hyun et al. (UM) Inverting the ratio September 2012 14 / 60

Page 17: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Mixed Initiative Nested Classification

Recent achievements by numbers (year 5)

1 accepted and 3 submitted journal papers

7 accepted or submitted conference papers

Co-organizer and session chair for an invited session on “informationcollection and decision making” for ACC’12

B. Hyun et al. (UM) Inverting the ratio September 2012 15 / 60

Page 18: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Theoretical background

What is a classifier?

A decider D is a deterministic mapping defined on a set of data intotruth values

D : {data} → {T, F}A classifier C is a decider with the domain of the mapping being aspecific realization of a random variable

The difference between a decider and a classifier is that the latteraccounts for the randomness of the data being classified

Important parameters

Processing of the data requires two abilities1 recognizing truth out of truth (rate of true positives)2 recognizing falsehood out of falsehood (rate of true negatives)

Characterized by two independent parameters σT and σF

B. Hyun et al. (UM) Inverting the ratio September 2012 16 / 60

Page 19: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Theoretical background

Theoretical background - Probabilistic modeling

Let X ∈ {T, F} be the object category variable

Let Y ∈ {Y1, Y2} be the object property variable

The likelihood is modeled by the following conditional probabilities,

P (Y = Y2|X = T ) = σT ,

P (Y = Y1|X = F ) = σF ,

P (Y = Y1|X = T ) = 1− σT ,P (Y = Y2|X = F ) = 1− σF , (1)

where σi ∈ [0.5, 1], i ∈ {T, F}.u: proportion of sub-population T among the entire population

B. Hyun et al. (UM) Inverting the ratio September 2012 17 / 60

Page 20: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Theoretical background

Theoretical background - Maximum likelihood classification

Bayes rule

Provides posterior probability of the object category on the basis theobject property

P (X = T |Y = {Y1, Y2}) =P (Y = {Y1, Y2}|X = T )P (X = T )

P (Y = {Y1, Y2})(2)

Let Os ∈ {T, F} be the decision variable

Likelihood-ratio rule

Makes classification decisions by comparing the posterior probability

Os =

{T if P (X=T |Y={Y1,Y2})

P (X=F |Y={Y1,Y2}) > λ

F if P (X=T |Y={Y1,Y2})P (X=F |Y={Y1,Y2}) ≤ λ.

(3)

where λ ∈ R.

B. Hyun et al. (UM) Inverting the ratio September 2012 18 / 60

Page 21: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Theoretical background

Theoretical background - Classification performance

Probability of misclassification

The performance measure is the probability of misclassification, Pm,

Pm = P (Os = T ∧X = F ) + P (Os = F ∧X = T ) (4)

Pm is the sum of probabilities of two faulty outcomes: the probabilityof false positives and the probability of false negatives

B. Hyun et al. (UM) Inverting the ratio September 2012 19 / 60

Page 22: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Theoretical background

Thresholding problem

Assumptions

A continuous measurable property w ∈ RObject categories are known a priori

Distribution of w for each object category is known a priori

w F T €

σF

σT

FN

FP

Figure: Dichotomous thresholding

w  F   T  €

σF

σT

FN

FP

Unknown  

Figure: Trichotomous thresholding

B. Hyun et al. (UM) Inverting the ratio September 2012 20 / 60

Page 23: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Theoretical background

Mixed-initiative nested thresholding

Start

End

Prior

Workload-IndependentTrichotomous Classifier

Workload-DependentDichotomous Classifier

Good?

u

T, F Pm

T, F Pm

WundecidablesYes

No

B. Hyun et al. (UM) Inverting the ratio September 2012 21 / 60

Page 24: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Theoretical background

Mixed-initiative nested thresholding

Start

End

Prior

Workload-IndependentTrichotomous Classifier

Workload-DependentDichotomous Classifier

Good?

u

T, F Pm

T, F Pm

WundecidablesYes

No

w F T !

"F

!

"T

!

FN

!

FP

!"!" #"!

"F

!

"T

!

FN

!

FP

$%&%'(%"

Workload

Workload

B. Hyun et al. (UM) Inverting the ratio September 2012 22 / 60

Page 25: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Theoretical background

Problem formulation

Workload

We define a workload variable, W ∈ [0, 1], with 0 indicating idle and 1indicating fully loaded. Let fi(w) = aie

−(w+bi)2/c2i with i ∈ {T, F}, then

the workload variable is defined as

W =

∫ τ2

τ1

ufT (w) + (1− u)fF (w)dw. (5)

Note that the range of W is [0, 1] for any τ1 and τ2.

The region of indecision, i.e., [τ1, τ2], of the primary trichotomousclassifier determines the workload applied to the secondary classifier.

Optimization problem

minτ1,τ2

P 2m,

subject to some inequality constraints.

B. Hyun et al. (UM) Inverting the ratio September 2012 23 / 60

Page 26: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Theoretical background

Comparison of performance

10−1

100

101

102

103

10−8

10−7

10−6

10−5

10−4

10−3

10−2

10−1

100

Cl

Pm*

Minimal probability of misclassification vs. classifiability

Pm2 with τ

0 = [m

T, m

F]

Pm1

(a) The minimal probability of misclassi-fication vs. classifiability. The blue solidline indicates the mixed-initiative nestedthresholding while the red dashed line in-dicates the dichotomous thresholding.

10−1

100

101

102

103

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Cl

W

Workload vs. Classifiability

W with τ0 = [m

T, m

F]

(b) Workload vs. classifiability

Figure: The minimal probability of misclassification and the workload for themixed-initiative nested thresholding as a function of the classifiability

B. Hyun et al. (UM) Inverting the ratio September 2012 24 / 60

Page 27: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Inverting the ratio

Inverting the ratio

H1 · · ·H2 H3 H10

M1 M1 M10M2

H1

· · ·

Figure: Mixed-initiative nested thresholding with more than two team members.(M denotes a workload-independent classifier and H denotes aworkload-dependent classifier)

Point of interest

Given a workload W provided by a workload-independent classifier (M),

What’s the optimal ratio of the mixed-initiative team members?

What’s the reachable performance?

What’s the individual workload applied to each workload-dependentclassifiers (H)?

B. Hyun et al. (UM) Inverting the ratio September 2012 25 / 60

Page 28: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Inverting the ratio

Setup

Ratio variable n ∈ { 1m ,

1m−1 , · · · , 1

2 , 1, 2, · · · ,m}the ratio of the number of workload-dependent classifiers to thenumber of workload-independent classifiers in the system with m ∈ N.n = 0.1 means a single workload-dependent classifier (human) and 10workload-independent classifiers (machines).

Total workload Wt ∈ [0, ∞)

the workload applied to the whole secondary layer in the architecture

Individual workload Wn ∈ [0, 1], Wn = Wtn

the workload applied to the individual classifier in the secondary layerassume uniform distribution of Wt to the secondary layer

Problem formulation

The objective of the problem is to minimize the probability ofmisclassification by choosing the ratio number n, i.e.,

minnP 2m(Wt, n).

B. Hyun et al. (UM) Inverting the ratio September 2012 26 / 60

Page 29: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Inverting the ratio

Analytical results

Theorem

Suppose that Wt is fixed and let n∗ = arg minn P2m(Wt, n). The optimal

ratio n∗ is monotonically increasing with respect to Wt, specifically thatn∗ = 2Wt.

†Proof by the necessary condition for optimality

Corollary

limWt→∞

W ∗n = 0.5

B. Hyun et al. (UM) Inverting the ratio September 2012 27 / 60

Page 30: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Inverting the ratio

10−1

100

0.20.25

0.33

0.5

1

2

Total workload (W)

Opt

imal

rat

io (

n* )

(a) Optimal ratio (abscissain logarithmic scale)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.01

0.02

0.03

0.04

0.05

0.06

Total workload (W)

Min

imal

pro

babi

lity

of m

iscl

assi

ficat

ion

(Pm2

*(n* ))

(b) Minimal probability ofmisclassification

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

Total workload (W)

Indi

vidu

al w

orkl

oad

(Wn)

(c) Individual workload

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

1

2

3

4

5

6

7

8

9

10

X: 0.1Y: 0.2 Total workload (W)

Opt

imal

rat

io (

n* )

X: 0.2Y: 0.3333

X: 0.3Y: 0.5

(d) Optimal ratio

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

Total workload (W)

Min

imal

pro

babi

lity

of m

iscl

assi

ficat

ion

(Pm2

*(n* ))

(e) Minimal probability ofmisclassification

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

0.35

0.4

0.45

0.5

0.55

0.6

0.65

Total workload (W)

Indi

vidu

al w

orkl

oad

(Wn)

(f) Individual workload

B. Hyun et al. (UM) Inverting the ratio September 2012 28 / 60

Page 31: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Conclusion

Conclusion

Implications

Guidelines to design a mixed-initiative system that autonomouslydetermines the optimal human-to-machine ratio

Relevant publications - available in MACCCS Ctools website1 B. Hyun, M. Faied, P. Kabamba, A. Girard, Mixed-Initiative Nested Classification for n Team Members, IEEE

Conference on Decision and Control, Maui, HI, 2012.

2 B. Hyun, M. Faied, P. Kabamba, A. Girard, Optimal Multivariate Classification by Linear Thresholding, AmericanControl Conference, Montreal, Canada, 2012. (invited paper)

3 B. Hyun, M. Faied, P. Kabamba, A. Girard, Optimal Classification by Mixed-Initiative Nested Thresholding, IEEETransactions on Systems, Man, and Cybernetics - Part A, 2012, Submitted.

4 B. Hyun, M. Faied, P. Kabamba, A. Girard, On Minimizing Classification Error by Maximizing Information, IEEE SignalProcessing Letters, 2012, Submitted.

B. Hyun et al. (UM) Inverting the ratio September 2012 29 / 60

Page 32: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Conclusion

Optimal strategies for team classification

Problem

Given: a number of decision makers, their individual performancesand prior information.

Find: the best fusion rules under different decision structures withrespect to a performance metric.

A1 B1

A2

B2

A3

B3

(g) Incremental pairing

A1 B1

A3

A2 B2

B3

(h) Tournament-like pairing

B. Hyun et al. (UM) Inverting the ratio September 2012 30 / 60

Page 33: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Conclusion

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.05

0.1

0.15

0.2

0.25

0.3

0.35

u

Min

imal

Pm

The misclassification of four−team classifier with incremental pairing

Fused Result for A2Fused Result for A3Final Fused Result

(i) Incremental pairing

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.05

0.1

0.15

0.2

0.25

0.3

0.35

u

Min

imal

Pm

The misclassifaction of Four−team classifier with Touramnet−like Pairing

Fused Result for B3Fused Result for A3Final Fused Result

(j) Tournament-like pairing

We propose a decision structure that exploits a moderator, i.e., anentity that exploits Bayesian inference from individual classifiers’decisions and makes final decisions based on maximum likelihoodclassification.

Two pairing schemes, i.e., incremental and tournament-like, areproposed and we show that the incremental pairing is the mosteffective decision structure among the proposed ones.

S. Pan, B. Hyun, P. Kabamba, A. Girard, Optimal Fusion Rules in Team Classification under Three Decision Structures,

American Control Conference, Washington, DC, USA, 2013, Submitted.

B. Hyun et al. (UM) Inverting the ratio September 2012 31 / 60

Page 34: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Conclusion

Future work

Analysis under different performance measures

- Addressing time-criticality by queueing theory- Confidence level

Kinematic classification (free measurements)

- Costly kinematic classification (costly measurements)

Classification with learning

Strategies for uncertain prior information

Deceptive strategies

B. Hyun et al. (UM) Inverting the ratio September 2012 32 / 60

Page 35: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Conclusion

Automated Classification Systemfor Bone Age X-ray Images

Jinwoo Seok, Baro Hyun, Josephine Kasa-Vubu*, and Anouck Girard

Department of Aerospace Engineering and Pediatric Endocrinology*University of Michigan, Ann Arbor, MI

Annual MACCCS Review

September 2012

J. Seok et al. (UM) Automated Classification System September 2012 33 / 60

Page 36: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Introduction

Motivation

Hand X-ray Image

Importance of Bone Age(BA)

The assessment of growth and pubertalmaturation is central to the practice ofpediatric endocrinology and BA is keyreferenceGreulich and Pyle (GP) atlas is a keyclinical indicator in pediatric endocrinologyTo determine BA, radiologist compares thepatient’s x-ray to those contained in thereference atlas and determines which imagein the atlas the patient’s x-ray is closest to

J. Seok et al. (UM) Automated Classification System September 2012 34 / 60

Page 37: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Introduction

Literature Review

There have been attempts at automated BA detection

CASAS [Tanner ’92]Peitka [Pietka et al. ’01]BoneXpert [Thodberg et al. ’01 and ’09]

BoneXpert has been developed recently

Active Appearance Model (AAM) [Cootes et al. ’01]Better performance than previous work [Martin et al. ’09](Root mean square deviation 0.72 years)

Problems of BoneXpert

Validating problemsClinical Age (CA) and BA relationship is unclear from the publications

J. Seok et al. (UM) Automated Classification System September 2012 35 / 60

Page 38: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Introduction

Original Contributions

Image  morphing  

Greulich  and  Pyle  (1959)  

Radiographic  data  

More  radiographic  data  

−100 −80 −60 −40 −20 0 20 40 60 80 1000

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

v

f(v)

u = 0.5

ufT(w), mT = −10, sT = 10(1−u)fNT(w), mNT = 10, sNT = 15ufT(w)+(1−u)fNT(w)

Optimal Threshold

Thresholding  classifier  

Predicted  bone  age!  

Bone  age?  

Feature  extrac=on  

Training  

Schematic overview of the automated classification system

i. Create a modified atlas that hasimages regularly spaced atthree month intervals in theclinically significant ranges

ii. Propose a novel Singular ValueDecomposition (SVD)-basedfeature extractor to create afeature vector out of thedescriptors obtained from SIFT

iii. Develop image classifier basedon SIFT - SVD

J. Seok et al. (UM) Automated Classification System September 2012 36 / 60

Page 39: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Technical Section

Image Feature Extraction

0 100 200 300 400 500 600

100

200

300

400

500

600

Feature descriptors using VL-SIFT

Scale Invariant Feature Transform (SIFT)

Introduced by David G. Lowe in 1999

Local-based feature extraction method

Invariant to scaling and rotation, andpartially invariant to viewpoint andillumination changes

Algorithm

Detection of scale-space extremaAccurate keypoint localizationOrientation assignmentThe local image descriptor

J. Seok et al. (UM) Automated Classification System September 2012 37 / 60

Page 40: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Technical Section

Image Feature Extraction

Singular Value Decomposition (SVD)

Matrix factorization method

Reduces the size while keeping the characteristics of a matrix

Given an m×m matrix A, the expression of its SVD is

A = UΣV T (6)

where U is an m×m matrix, V is an n× n matrix and Σ is the singular values of matrix

A which is an m× n non-negative real diagonal matrix.

SIFT - SVD based feature extractor

By applying SVD to the feature descriptors obtained from SIFT, weproduce a novel feature vector for the classifier.

J. Seok et al. (UM) Automated Classification System September 2012 38 / 60

Page 41: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Simulation

Simulation

Data set

24 GP female standard images for training: 1 through 27 excluding 13,21 and 27, 13 and 27 because of poor image conditions.Generated 19 morphing images for validation.

Classification decision step

Import images to MatlabApply the SIFT algorithm to get key points and local image descriptorsApply SVD to get reduced feature vectorsTrain the neural networkValidateIn progress: gathering larger data set for statistical analysisFuture work: compare Hyun approach to current (neural network)

J. Seok et al. (UM) Automated Classification System September 2012 39 / 60

Page 42: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Simulation

Results

Test result 1, marked with circles

Classifier works well as most theanswers are closely aligned to thediagonal line.

Only one result shows radicalmisclassification.

Three results showing moderateerrors, and some round-off errors.

Test result 2, marked with crosses

Classifier performs less well: Therewas only one training data per class;this is generally not consideredsufficient to train classifiers. (Proof ofconcept).

0 5 10 15 20 250

5

10

15

20

25

Input (GP standard number)

Ou

tpu

t (G

P s

tan

da

rd n

um

be

r)

Correct Answer

Test result1

Test result2

SIFT - SVD classifier results

J. Seok et al. (UM) Automated Classification System September 2012 40 / 60

Page 43: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Simulation

Highlights of Other Relevant Research

Justin Jackson, Eric Sihite, Ricardo Bencatel

Annual MACCCS Review

September 2012

ARC Lab Team (UM) Other Research September 2012 41 / 60

Page 44: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Relevant Research

Highlights of Other Relevant Research

Distributed Task Assignment and Scheduling

VRP Heuristics Comparison

Persistent Flight on Flow Fields

ARC Lab Team (UM) Other Research September 2012 42 / 60

Page 45: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Relevant Research

Task Assignment and Scheduling: Original Contributions

Contributions in two categories

Centralized minimum-time, precedence-constrained, vehicle routing

Distributed minimum-time, constrained, task assignment and taskscheduling

ARC Lab Team (UM) Other Research September 2012 43 / 60

Page 46: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Relevant Research

Centralized Task Assignment and Scheduling

Minimum-time, precedence-constrained vehicle routing

1 Low complexity algorithm for AFRL-relevant vehicle routing problem

2 Analysis of algorithm optimality and complexity

3 Solution quality measurement technique, useful in absence ofanalytical bounds

Comparison of tabu/2-opt heuristic and optimal tree search method for assignment problems, International Journal of

Robust and Nonlinear Control, 2011

A New Measure of Solution Quality for Combinatorial Task Assignment Problems, Conference on Decision and Control,

2010

A Combined Tabu Search and 2-opt Heuristic for Multiple Vehicle Routing, Automatic Controls Conference, 2010

ARC Lab Team (UM) Other Research September 2012 44 / 60

Page 47: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Relevant Research

Distributed Task Assignment and Scheduling

Minimum-time constrained distributed task assignment and scheduling

1 Communication-constraints satisfy operational needs

2 Scheduling constraints express relevant operational constraints

3 Stochastic Bidding and the OptDNSB Algorithms for assignment andscheduling

4 Correctness, completeness, optimality, complexity characterization

5 Characterization and utilization of problem separation

Distributed Constrained Minimum-Time Schedules in Networks of Arbitrary Topology, IEEE Transactions on Robotics,

2011 (Submitted)

Communication-Constrained Distributed Assignment on Networks of Arbitrarily Topology, IEEE Transactions on

Robotics, 2011 (Submitted)

Communication-Constrained Distributed Assignment, IEEE Conference on Decision and Control, 2011

Distributed Task Scheduling Subject to Arbitrary Constraints, 18th World Congress of the International Federation of

Automatic Control (IFAC), 2011

ARC Lab Team (UM) Other Research September 2012 45 / 60

Page 48: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Relevant Research

Heuristics Comparison for VRP

ARC Lab Team (UM) Other Research September 2012 46 / 60

Page 49: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Relevant Research

Heuristics Comparison for VRP

ARC Lab Team (UM) Other Research September 2012 47 / 60

Page 50: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Relevant Research

Heuristics Comparison for VRP

ARC Lab Team (UM) Other Research September 2012 48 / 60

Page 51: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Relevant Research

Heuristics Comparison for VRP

E. Sihite, J. Jackson, A. Girard, VRP Heuristics Comparison, ACC 2013 (Submitted)

ARC Lab Team (UM) Other Research September 2012 49 / 60

Page 52: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Relevant Research

Perpetual Flight in Flow Fields

Extension of UAV endurance

Inspired by birds behaviors

Harvest airflow energy

ARC Lab Team (UM) Other Research September 2012 50 / 60

Page 53: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Thermal Soaring

Models - Chimney & Bubble ThermalsObservability

Estimation

(m) Leaning ChimneyThermal

(n) Bubble Thermal

ARC Lab Team (UM) Other Research September 2012 51 / 60

Page 54: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Thermal Soaring

ModelsObservabilityEstimation

(o) Trapezoidal model

Theorem: The thermal position, velocity and updraft flow field planar parameters are

locally weakly observable by an aircraft flying trajectories with ϕ̇ 6= γ̇ tan2 (ϕ− γ), as

long as the trajectory is included in the area defined by r2 ≥ d ≥ r1. This holds for the

trapezoidal model.

The aircraft cannot fly at a constant distance from the thermalcenter.The aircraft should be flying around the thermal or turning

ARC Lab Team (UM) Other Research September 2012 52 / 60

Page 55: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Thermal Soaring

Models

Observability

Estimation - Regularized Adaptive Particle Filter

(p) Estimator initialization

ARC Lab Team (UM) Other Research September 2012 53 / 60

Page 56: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Thermal Soaring

Models

Observability

Estimation - Regularized Adaptive Particle Filter

(q) Estimator convergence

ARC Lab Team (UM) Other Research September 2012 54 / 60

Page 57: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Wind Shear Soaring

Models - Surface, Layer & Ridge Wind ShearEstimation

(r) Surface Wind Shear

ARC Lab Team (UM) Other Research September 2012 55 / 60

Page 58: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Wind Shear Soaring

Models - Surface, Layer & Ridge Wind ShearEstimation

(s) Layer Wind Shear (t) Ridge Wind Shear

ARC Lab Team (UM) Other Research September 2012 56 / 60

Page 59: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Wind Shear Soaring

ModelsEstimation - Particle Filter

(u) Estimator initialization

ARC Lab Team (UM) Other Research September 2012 57 / 60

Page 60: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Wind Shear Soaring

ModelsEstimation - Particle Filter

(v) Estimator final convergence

ARC Lab Team (UM) Other Research September 2012 58 / 60

Page 61: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Formation Flight

Validation of airflow modelsCollection of spatially distributed samplesSafe flight at close distances

(w) Formation in a thermalARC Lab Team (UM) Other Research September 2012 59 / 60

Page 62: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Formation Flight

Validation of airflow modelsCollection of spatially distributed samplesSafe flight at close distances

(x) Formation in a thermalARC Lab Team (UM) Other Research September 2012 59 / 60

Page 63: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Formation Flight

Validation of airflow modelsCollection of spatially distributed samplesSafe flight at close distances

(y) Formation in a thermalARC Lab Team (UM) Other Research September 2012 59 / 60

Page 64: MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM · MAX: Collaborative Unmanned Air Vehicles Recent Progress at UM Anouck Girard & Pierre Kabamba Baro Hyun, Justin Jackson,

Perpetual Flight in Flow Field

Thank You!

ARCLAB (UM) Collaborative Unmanned Air Vehicles 60 / 60