View
45
Download
3
Category
Preview:
DESCRIPTION
Recognition System Capacity. Joseph A. O’Sullivan Samuel C. Sachs Professor Electronic Systems and Signals Research Laboratory Department of Electrical and Systems Engineering Washington University in St. Louis (314) 935-4173; http://essrl.wustl.edu/~jao jao@wustl.edu - PowerPoint PPT Presentation
Citation preview
Recognition System CapacityJoseph A. O’Sullivan
Samuel C. Sachs ProfessorElectronic Systems and Signals Research Laboratory
Department of Electrical and Systems EngineeringWashington University in St. Louis
(314) 935-4173; http://essrl.wustl.edu/~jao jao@wustl.edu
Michael D. DeVore, UVANaveen Singla
Brandon WestoverSupported in part by the Office of Naval Research
Adaptive Sensing MURI Review, 06/27/06
2
O’Sullivan AS-MURI 2006
Recognition System Capacity
• Motivation: – ATR; Network Centric Warfare;
Biometrics; Image Understanding• Active Computations• Achievable Rate Regions
– Inner and outer bounds– Successive refinement
3
O’Sullivan AS-MURI 2006
Why Theorems?• ONR Perspective: Want Systems
That Work– Implementable on projected system
architecture– Good performance
• Our Perspective: Theorems Provide– Provable performance: bounds and
guidelines– Validation and critique of existing
system designs– Motivation for recognition system
design: system architectures; database design; optimal compression for recognition; communication for recognition; active computations
• Growing Awareness of Importance of Information Theory
4
O’Sullivan AS-MURI 2006
Perspective on Image Understanding
• “Finding tanks is so World War II.” Bob Hummel, DARPA program manager, ATR Theory Workshop, Dec. 2004– What make of car?– What year?– Who is driving?– Where has it been?
• Improvised Explosive Devices (IED)• Demand more information from imagery
5
O’Sullivan AS-MURI 2006
Biometrics Must Be
• Universal• Permanent• Unique• Measurable
Uniqueness How unique? Bits.Measurability How measurable? Bits.
A. K. Jain, et al., “Introduction to Biometrics,” 1999John Daugman, http://www.cl.cam.ac.uk/users/jgd1000/
Encoding
6
O’Sullivan AS-MURI 2006
Recognition System Capacity
• Motivation: – ATR; Network Centric Warfare; Biometrics;
Image Understanding• Active Computations• Achievable Rate Regions
– Inner and outer bounds– Successive refinement
7
O’Sullivan AS-MURI 2006
Sensor Data
Target Type Estimate
Performance Estimate
Algorithm(ATR)Sensors
Network Resources databases,communications, etc.
System Performance Analysis
ResourceAllocation
Active Computations Concept
Compute a sequence of inferences and performance estimates (probabilities or reliabilities). Monitor available resources (time, processors, bandwidth, database, …). Feed back performance: select next computation; reallocate resources; demand more data.
8
O’Sullivan AS-MURI 2006
Sensor Data
Target Type Estimate
Performance Estimate
Algorithm(ATR)Sensors
Network Resources databases,communications, etc.
System Performance Analysis
ResourceAllocation
Active Computations Concept
Successively refined inferences Time or resources to achieve performance goal. Additional data required to achieve performance goal.
9
O’Sullivan AS-MURI 2006
Total System PerformanceResource Consumption Approximations
• ATR system performance entails more than just accuracy:– Time to classify a target– Electrical power dissipation– Sensor engagement, CPU cycles, bits communicated, and other
“opportunity costs”
• Need real-time estimates of total system performance– Enable informed tradeoff of ATR accuracy with throughput and network
resource consumption– Dynamically adapt the system as requirements, capabilities, and
operational scenarios evolve
12
O’Sullivan AS-MURI 2006
Active Computation• The need to actively manage computations is
acute in complex, time-critical environments– Information has a time value– Some information now may be better than a lot of information later,
after it is too late to take decisive action– Ideally, we’d like some information now and more later
• Static ATR implementations perform the same computations for every image they receive– No tentative answers are available before processing is finished– Availability of more time will not improve the solution accuracy
13
O’Sullivan AS-MURI 2006
Active Computation• Dynamic ATR systems employ active
computations to maximize the time-value (or resource-value) of information
• Approach: Generate a sequence of increasingly accurate classifications– More resources are consumed at every stage– Continue until accuracy is good enough, resource cap is reached,
or the result is no longer relevant– Control the computations to maximize the total information value
14
O’Sullivan AS-MURI 2006
• Maximum-likelihood ATR solution is
• Solve a sequence of simpler problems– The functions get closer to with each stage– Each problem is easy to solve given previous solutions– Let be the sequence of problems that are chosen up to stage k– Let be the error and be the resources used– The best strategy at stage K minimizes the total expected cost
Active Computation
15
O’Sullivan AS-MURI 2006
• Seek heuristic strategies that do not require prior knowledge of K, but are nearly optimal for all K
• For example, maximize the expected future increase in likelihood
Active Computation
16
O’Sullivan AS-MURI 2006
Recognition System Capacity
• Motivation: – ATR; Network Centric Warfare; Biometrics;
Image Understanding• Active Computations• Achievable Rate Regions
– Inner and outer bounds– Successive refinement
17
O’Sullivan AS-MURI 2006
Recognition System Capacity:More Motivation
● Number of bits for recognition● Number of patterns that can be distinguished● Number of bits to extract from data● Size of long term memory● Data and processing dependence● Start with simple i.i.d. model
18
O’Sullivan AS-MURI 2006
X1
X2
.
.
.XMc
Selectone
p(x) X1
X2
.
.
.XMc
p(y|x)
y
Selectone
p(h)
xh
Objective: Pr{g(V)=h}>1-ε, s.t. R=(Rc, Rx, Ry)
U1,U2,…,UMx
fmemoryencoder
gh
φsensoryencoder
V
memoryrepresentation
sensoryrepresentation
Rc
Rx Ry
19
O’Sullivan AS-MURI 2006
Pattern Recognition Codes and Achievable Rates
p(x)
x1 x2 x3
xMc
… p(y|x)
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
xh yselect patternp(h)
g2
…
y
f g1
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
p(x)
x1 x2 x3
xMc
… p(y|x)
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
xh yselect patternp(h)
select patternp(h)
g2
………
y
f g1
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
p(x)
x1 x2 x3
xMc
… p(y|x)
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
xh yselect patternp(h)
…
y
f
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
p(x)
x1 x2 x3
xMc
… p(y|x)
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
xh yselect patternp(h)
select patternp(h)
………
y
f
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
p(x)
x1 x2 x3
xMc
… p(y|x)
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
xh yselect patternp(h)
g2
…
y
g1
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
p(x)
x1 x2 x3
xMc
… p(y|x)
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
xh yselect patternp(h)
select patternp(h)
g2
………
y
g1
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
p(x)
x1 x2 x3
xMc
… p(y|x)
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
xh yselect patternp(h)
…
y
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
p(x)
x1 x2 x3
xMc
… p(y|x)
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
(i)i
v(i)
11…11My
…00…1000…0100...00…321
…
xh yselect patternp(h)
select patternp(h)
………
y
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
(i)i
u(i)
11…11Mx
…00…1000…0100...00…321
…
20
O’Sullivan AS-MURI 2006Characterizing
21
O’Sullivan AS-MURI 2006
0
I(X;U)
I(Y;V)
H(Y)
H(X)
0
Rx > I(X;U)Ry > I(Y;V)Rc < I(U;V)-I(U;V|X,Y)
On the border,
U-X-Y-V , so
R*=R**=R.
V=Y
U=X
Rc=I(X;Y)
`Unlimited’ U,V capacity: U=X, Y=V
Rc < I(X;Y)Random channel coding
Rc=0
Rc=0
Poor memory: U=0
Rc < I(0;V)=0
Poor senses:V=0
Rc<I(U;0)=0.
Rc=I(X;Y)-I(X;Y|U)
Rc=I(X;Y)-I(X;Y|V)
22
O’Sullivan AS-MURI 2006
f
g
φ
(U,V)
X
Yp(x,y)
A Related Gap: the distributed source coding problem
- Problem: Characterize the achievable (Rx,Ry,Dx,Dy)-Sergio Servetto claimed solution at ITW 2006-Solution should transfer to our problem
23
O’Sullivan AS-MURI 2006
Related Work
25
O’Sullivan AS-MURI 2006
Sensor Data Link
Weapon LinkShooter Link
•Recce Imagery (SAR, IR, Visible)•Intelligent Bandwidth Compression
Ground Recce/Intel Station Strike Planning System
•Wide Area Cueing (ATC)
•Select Target
Targeting Info Link•Link Target data to TOC(type, location, motion,…)
•Strike Planning•Weapon Selection
•ATR•Reference Library, Aimpoint
Terminal ATR
•Strike plan•Target location•ATR parameters
Naval Impact/Payoff: The Sensor to Shooter Problem
26
O’Sullivan AS-MURI 2006
Convex Hull Inner BoundWestover and O’Sullivan ISIT 2005
• The convex hull of this inner bound is achievable.• Consider the set of all distributions such that conditioned
on a random variable Q, we have U – X – Y – V. Then the achievable region is
• For every case examined, this convex hull is achieved by time sharing between a length 4 MC and the (0,0,0) point.
)|;(
)|;()|;(
QVUIR
QVYIRQUXIR
c
y
x
),,( 1111 cyx RRRR
}2,,2,1{}2,,2,1{:
}2,,2,1{:
}2,,2,1{:
11
1
1
1
1
1
cy
y
x
nRnR
nRn
nRn
g
Y
Xf
),,( 2222 cyx RRRR
}2,,2,1{}2,,2,1{
}2,,2,1{}2,,2,1{:
}2,,2,1{:
}2,,2,1{:
)(
)(2
)(2
)(2
121
121
12
12
ccc
yyy
yy
xx
RRnnR
RRnnR
RRnn
RRnn
g
Y
Xf
Coarse Code: (f1,Φ1,g1)n Refining Code: (f2,Φ2,g2)n
The rate sextuplet (Rx1,Ry1,Rc1,Rx2,Ry2,Rc2) is achievable if there exist sequences of recognition codes (f1,Φ1,g1)n and (f2,Φ2,g2)n such that Comment: two different systems (different patterns)
.0, 21 ne
ne PP
Successive Refinement, Two-Stage RecognitionGiven a sequence of (Mx1, My1, Mc1, n) pattern recognition codes, design a sequence of (Mx2, My2, Mc2, n) PR codes with the first sequence as subcodesMx1≤ Mx2 My1 ≤ My2 Mc1 < Mc2
“Up and to the right”
Achievability: Inner Bound
);(
);();(
111
11
11
VUIR
VYIRUXIR
c
y
x
),;,(
),;(),;(
21212
212
212
VVUUIR
VVYIRUUXIR
c
y
x
Theorem: Two-stage recognition is achievable if there existauxiliary random variables U1, V1, U2, and V2 satisfying• Markov conditions: U1 – X – Y – V1 and
(U1,U2) – X – Y – (V1,V2).• Rate Bounds:
Inner Bound: Proof Sketch. At the coarse stage:•Use the coding strategy for the single-stage pattern recognition systemAt the refining stage:•Given memory and sensory indices from the coarse stage, generate “refining” codebooks according to the conditional distributions p(u2|u1(·)) and p(v2|v1(·)).•Encode memory and sensory data with pairs of indices corresponding to coarse and refining stage•Use typical-set decoding to identify pattern
29
O’Sullivan AS-MURI 2006
Inner Bound: Successive Refinement
);(
);();(
111
11
11
VUIR
VYIRUXIR
c
y
x
);(
);();(
222
22
22
VUIR
VYIRUXIR
c
y
x
Analogous to the Markov condition for successive refinement for rate-distortion.
Equitz and Cover, “Successive refinement of information,” IEEE Trans. Info. Theory, Mar. 1991.
Corollary: Successive refinement is achievable if there existauxiliary random variables U1, V1, U2, and V2 satisfying• Markov condition: U1 – U2 – X – Y – V2 – V1. • Rate bounds:
30
O’Sullivan AS-MURI 2006
Converse: Outer BoundTheorem: If the rate sextuplet (Rx1,Ry1,Rc1,Rx2,Ry2,Rc2) isachievable then there exist auxiliary random variables U1,V1, U2, and V2 satisfying• Markov conditions: U1 – X – Y and X – Y – V1 and
(U1,U2) – X – Y and X – Y – (V1,V2). • Rate Bounds:
),|;();(
);();(
11
111
11
11
YXVUIVUIR
VYIRUXIR
c
y
x
),|,;,(),;,(
),;(),;(
2121
21212
212
212
YXVVUUIVVUUIR
VVYIRUUXIR
c
y
x
Corollary: Two length 4 Markov chains follow: U1 – U2 – X – Y and X – Y – V2 – V1
31
O’Sullivan AS-MURI 2006
Extension: Hierarchical Recognition Based on Random Labels
• Extend results so that the codebook is the same for the two stages
• Randomly label each Xn(k) with a label L(k), out of exp[n(Rc2-Rc1)] labels
• For each label, use a (Mx1, My1, Mc1, n) pattern recognition code
• Given Yn, run every decoder (for every label) list of exp[n(Rc2-Rc1)] possible patterns
• Use refinement codebooks to determine label and therefore the pattern
32
O’Sullivan AS-MURI 2006
Extension: Hierarchical Recognition Based on Hierarchical Pattern Model
• Assume that the patterns are generated by a hierarchical model W X (class identity)
• Inner Bound: U1 – W – Y – V1 and U1 – U2 – X – Y – V2 – V1
• Use a (Mx1, My1, Mc1, n) pattern recognition code to obtain Wn(i) (class)
• Use refinement codebooks to determine Xn(i,j) (identity)
))(|( i.i.d. 2 ..., ,2 ,1 ),,(
)( i.i.d. 2 ..., ,2 ,1 ),()( 12
1
iwxpjjiX
wpiiW
kRRnn
nRn
cc
c
p(w)
)1,1(nX
)1(nW
)2(nW
)2( 1cnRnW
)2,1(nX
)2,1( )( 12 cc RRnnX
)1,2(nX)2,2(nX
)2,2( )( 12 cc RRnnX
)1,2( 1cnRnX)2,2( 1cnRnX
)2,2( )( 121 ccc RRnnRnX
p(x|w)
p(x|w)
p(x|w)
33
O’Sullivan AS-MURI 2006
Extensions• Inner bounds for prototypical examples:
Gaussian, binary. Convex hull is achievable by successive refinement.
• Successive refinement “up and to the right”
34
O’Sullivan AS-MURI 2006
Recognition System Design• Developing robust ATR algorithms, deriving
limits on recognition performance• Quantifying recognition performance as a
function of system resource measures• Developing algorithms and implementations
that adapt to dynamically varying resource constraints
– Time, availability of processors, communication bandwidth, data storage, sensor image quality
• Impact: increase efficiency and effectiveness of system implementations
– Information latency problem– Recognition systems using visual
imagery, SAR, ladar– Increase in ATR performance– Allow more imagery to be screened– Provide systematic tools for analyzing
design choices such as processors and network communication
Collaborators Washington UniversityJoseph A. O’SullivanAndrew LiNaveen SinglaPo-Hsiang LaiLee Montagnino Brandon WestoverRobert PlessRonald S. IndeckNatalia A. Schmid (UWVa)Michael D. DeVore (UVa)Alan Van Nevel
35
O’Sullivan AS-MURI 2006
36
O’Sullivan AS-MURI 2006
Selected Limitations of Existing Systems
• “Stovepipe” design– Fixed inputs, processing, database, output– Fixed time– Algorithms are not transparent
Seek “any-time” adaptive system design
37
O’Sullivan AS-MURI 2006
Naval Capability Provided“ Network centric warfare is military
operations that exploit information and networking technology to integrate widely dispersed human decision makers, situational and targeting sensors, and forces and weapons into a highly adaptive, comprehensive system to achieve unprecedented mission effectiveness.” Network-Centric Naval Forces, Naval Studies Board, National Research Council, 2000
Active Computations Exploit technology Integrate sensors, resource allocation,decision makers, algorithms Adapt to dynamically varying resources Provide measures of uncertainty as aa function of available resources
Recommended