Tracking of Animals Using Airborne Camerasusers.isy.liu.se/en/rt/clave42/presentations/presentation_lic2016.pdf · Licentiate’sPresentation ClasVeibäck November22,2016 3 Introduction

Tracking of AnimalsUsing Airborne CamerasClas Veibäck

1 Introduction2 Target Tracking3 Camera Sensor4 Constrained Motion Model5 Uncertain Timestamp Model6 Mode Observations7 Conclusions


Licentiate’s Presentation Clas Veibäck November 22, 2016 3

Introduction

• Tracking of animals

• Overhead cameras

• Contributions

• Applications


Introduction



• Contributions

• Applications


Introduction



• Contributions

• Applications


Introduction



• Contributions

• Applications


Main Contributions

Uncertain Timestamps

Constrained Motion Model Mode Observations

M1

M2

M1

M2

M1

M2

k− 1 k k+ 1


Main Contributions


Constrained Motion Model

Mode Observations

M1

M2

M1

M2

M1

M2

k− 1 k k+ 1


Main Contributions


Constrained Motion Model Mode Observations

M1

M2

M1

M2

M1

M2

k− 1 k k+ 1


Dolphin Application

• Dolphinarium at Kolmården

Wildlife Park

• Fisheye camera with

occlusions

• Reflections and changing

light conditions

• Constrained to basin


Dolphin Application


Wildlife Park


occlusions


light conditions



Dolphin Application


Wildlife Park


occlusions


light conditions



Dolphin Application


Wildlife Park


occlusions


light conditions



Bird Application

• Recording of birds in Emlen

funnels

• Detect take-off times

• Estimate take-off directions

• Stationary and flight modes


Bird Application


funnels





Bird Application


funnels





Bird Application


funnels





Orienteering Application

• GPS trajectory

• Control position known

• Improve position estimate



• GPS trajectory





• GPS trajectory





Target Tracking

Tracker

Presentation

Pre-processing Association TrackManagement

Estimation

Sensor Data


Pre-processing

• Raw Sensor Data

• Signal and Image Processing

• Detections

Tracker

Presentation


Estimation

Sensor Data


Pre-processing

• Raw Sensor Data


• Detections

Tracker

Presentation


Estimation

Sensor Data


Pre-processing

• Raw Sensor Data


• Detections

Tracker

Presentation


Estimation

Sensor Data


Association

• Targets

• Detections

• False and Missed Detections

• Tracks

• Hypothesis

• Multiple Hypotheses

Tracker

Presentation


Estimation

Sensor Data


Association

• Targets

• Detections


• Tracks

• Hypothesis


Tracker

Presentation


Estimation

Sensor Data


Association

• Targets

• Detections


• Tracks

• Hypothesis


Tracker

Presentation


Estimation

Sensor Data


Association

• Targets

• Detections


• Tracks

• Hypothesis


Tracker

Presentation


Estimation

Sensor Data


Association

• Targets

• Detections


• Tracks

• Hypothesis


Tracker

Presentation


Estimation

Sensor Data


Association

• Targets

• Detections


• Tracks

• Hypothesis


Tracker

Presentation


Estimation

Sensor Data


Track Management

• New track

• Confirmed track

• Dead track

Tracker

Presentation


Estimation

Sensor Data


Track Management

• New track

• Confirmed track

• Dead track

Tracker

Presentation


Estimation

Sensor Data


Track Management

• New track

• Confirmed track

• Dead track

Tracker

Presentation


Estimation

Sensor Data


Estimation

• Given association hypothesis

• Properties of each target

• Over time using model

Tracker

Presentation


Estimation

Sensor Data


Estimation




Tracker

Presentation


Estimation

Sensor Data


Estimation




Tracker

Presentation


Estimation

Sensor Data


Estimation

Linear Gaussian state-space model

xk = Fkxk−1 +wk, wk ∼ N (0,Qk),

yj = Hjxj + vj , vj ∼ N (0,Rj),

x0∼ N (x0,P0).

Posterior distribution

p(xk|x0,y1, . . . ,yk)

Tracker

Presentation


Estimation

Sensor Data


Estimation




x0∼ N (x0,P0).


p(xk|x0,y1, . . . ,yk)

Tracker

Presentation


Estimation

Sensor Data


Estimation




x0∼ N (x0,P0).


p(xk|x0,y1, . . . ,yk)

Tracker

Presentation


Estimation

Sensor Data


Estimation




x0∼ N (x0,P0).


p(xk|x0,y1, . . . ,yk)

Tracker

Presentation


Estimation

Sensor Data


Estimation




x0∼ N (x0,P0).


p(xk|x0,y1, . . . ,yk)

Tracker

Presentation


Estimation

Sensor Data


Estimation




x0∼ N (x0,P0).


p(xk|x0,y1, . . . ,yk)

Tracker

Presentation


Estimation

Sensor Data


Estimation




x0∼ N (x0,P0).


p(xk|x0,y1, . . . ,yk)

Tracker

Presentation


Estimation

Sensor Data



Camera Sensor

z

yx

p

xr

xc

yx

f


Camera Model

z

yx

p

xr

xc

yx

f

• Camera extrinsics

• Camera intrinsics

• Projection

• Perspective compensation

• Lens distortion


Camera Model

z

yx

p

xr

xc

yx

f



• Projection


• Lens distortion


Camera Model

z

yx

p

xr

xc

yx

f



• Projection


• Lens distortion


Camera Model

z

yx

p

xr

xc

yx

f



• Projection


• Lens distortion


Camera Model

z

yx

p

xr

xc

yx

f



• Projection


• Lens distortion


Dolphin - Camera Model z

yx

p

xr

xc

yx

f


Bird - Camera Model z

yx

p

xr

xc

yx

f


Foreground Segmentation

C D

A B

© 2015 IEEE

• Gaussian-mixture

background model

• Handles changing light

conditions

• Handles reflections

• Degree of confidence

z

yx

p

xr

xc

yx

f



C D

A B

© 2015 IEEE


background model


conditions



z

yx

p

xr

xc

yx

f



C D

A B

© 2015 IEEE


background model


conditions



z

yx

p

xr

xc

yx

f



C D

A B

© 2015 IEEE


background model


conditions



z

yx

p

xr

xc

yx

f






• Targets constrained to

region

• Feasible predictions

• Similar behaviour




region






region




Turning Model

ω(x) = dr(x)

∫N

(βd + βa

(p⊥ · l(n)

))w(x, n) dn

• Nearly constant speed

• Influence by edges

• Avoid collision with edges

• Align with edges

• Integrate along edge

• Preferred direction

• Polygon region


Turning Model

ω(x) = dr(x)

∫N

(βd + βa

(p⊥ · l(n)

))w(x, n) dn

x =

(pp

)=

xyxy







• Polygon region


Turning Model

ω(x) = dr(x)

∫N

(βd + βa

(p⊥ · l(n)

))w(x, n) dn

w(x, n) =1

‖p− n‖2







• Polygon region


Turning Model

ω(x) = dr(x)

∫N

(βd + βa

(p⊥ · l(n)

))w(x, n) dn







• Polygon region


Turning Model

ω(x) = dr(x)

∫N

(βd + βa

(p⊥ · l(n)

))w(x, n) dn







• Polygon region


Turning Model

ω(x) = dr(x)

∫N

(βd + βa

(p⊥ · l(n)

))w(x, n) dn







• Polygon region


Turning Model

ω(x) = dr(x)

∫N

(βd + βa

(p⊥ · l(n)

))w(x, n) dn

Clockwise

or

Counterclockwise







• Polygon region


Turning Model

ω(x) = dr(x)

N∑i=1

(βd + βa(p⊥ · li)

)wi(x)

l1lN

v1

v2

vN

©2015

IEEE







• Polygon region


Potential Field


Dolphin - Model Simulation


Dolphin - Model Comparisons

Detection regionConstraint regionConstant Velocity modelCoordinated Turn modelConstrained Motion model


Dolphin - Complete Framework



Uncertain Timestamp Model



• Traditional measurements

• Observations sampled at an

uncertain time

• Crime scene investigations

• Traces from animals





uncertain time







uncertain time







uncertain time





Consider a linear Gaussian state space model,


yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

j ),

x0 ∼ N (x0,P0),

Extend the model with





yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

j ),

x0 ∼ N (x0,P0),


z = Hzxτ + vz, vz ∼ N (0,Rz), τ ∼ p(τ).





yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

j ),

x0 ∼ N (x0,P0),







yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

j ),

x0 ∼ N (x0,P0),







yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

j ),

x0 ∼ N (x0,P0),







yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

j ),

x0 ∼ N (x0,P0),




Simple Uncertain Time Scenario

Consider the simple model,

xk = xk−1 + wk, wk ∼ N (0, Q),

yj = xj + vyj , vyj ∼ N (0, Ry)

for k ∈ {1, . . . , N} and two measurements y1 and yN .


z = xτ + vz, vz ∼ N (0, Rz), τ ∼ p(τ).



Consider the simple model,

xk = xk−1 + wk, wk ∼ N (0, Q),

yj = xj + vyj , vyj ∼ N (0, Ry)

for k ∈ {1, . . . , N} and two measurements y1 and yN .


z = xτ + vz, vz ∼ N (0, Rz), τ ∼ p(τ).



Time

0 0.2 0.4 0.6 0.8 1

Positio

n

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6Y

z (first case)z (second case)

The measurements are

• y1 = 0,

• yN = 1.

The two cases of

observations are

• z = 0.5 with flat prior.

• z = 1.5 with non-flatprior.



Time

0 0.2 0.4 0.6 0.8 1

Positio

n

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6Y



• y1 = 0,

• yN = 1.

The two cases of

observations are





Time

0 0.2 0.4 0.6 0.8 1

Positio

n

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6Y



• y1 = 0,

• yN = 1.

The two cases of

observations are




Posterior Distributions - Time

The posterior distribution of the uncertain time is

wτ , p(τ |Y, z)︸︷︷︸Posterior

∝ p(τ)︸︷︷︸Prior

N (z| zτ , Sτ )︸︷︷︸Likelihood

.







.







.



p(τ |Y, z)maxX p(X , τ |Y, z)p(τ)

τ

0 0.5 1

Pro

babili

ty

×10-3

0

0.5

1

1.5

2

τ

0 0.2 0.4 0.6 0.8 1

Pro

ba

bili

ty

×10-3

0

2

4

6

8


Posterior Distributions - States

The posterior distribution of the state is

p(xk|Y, z) =

N∑τ=1

wτ · N (xk|xτk,P

τk).




p(xk|Y, z) =

N∑τ=1

wτ · N (xk|xτk,P

τk).




p(xk|Y, z) =

N∑τ=1

wτ · N (xk|xτk,P

τk).




p(xk|Y, z) =

N∑τ=1

wτ · N (xk|xτk,P

τk).



Time

0 0.2 0.4 0.6 0.8 1

Po

sitio

n

-0.5

0

0.5

1

1.5

2

Time

0 0.2 0.4 0.6 0.8 1

Po

sitio

n

-0.5

0

0.5

1

1.5

2


Estimators - Minimum Mean Squared Error

Yz

XMMSE |Y, z

XMMSE |Y

Time

0 0.5 1

Positio

n

0

0.5

1

1.5

Time

0 0.5 1

Positio

n

0

0.5

1

1.5


Orienteering - Results

• Adjusted trajectory

• Single control point


Orienteering - Results

• Adjusted trajectory

• Single control point



Mode Observations

M1

M2

M1

M2

M1

M2

k− 1 k k+ 1


Mode Observations

M1

M2

M1

M2

M1

M2

k− 1 k k+ 1

• Jump Markov model

• Model target behaviour

• Probability of switching

• Direct mode observation


Mode Observations

M1

M2

M1

M2

M1

M2

k− 1 k k+ 1






Mode Observations

M1

M2

M1

M2

M1

M2

k− 1 k k+ 1






Mode Observations

M1

M2

M1

M2

M1

M2

k− 1 k k+ 1






Jump Markov ModelThe jump Markov linear model is

xk = Fk(δk)xk−1 +wk, wk ∼ N (0, Qk(δk)),

yj = Hj(δj)xj + vj , vj ∼ N (0, Rj(δj)).

The mode is modelled as

p(δk | δk−1) = Πδk,δk−1

k , δk ∈ S.

Augmented is the direct mode observation,

zi ∼ p(zi | δi).

M1

M2







k , δk ∈ S.


zi ∼ p(zi | δi).

M1

M2







k , δk ∈ S.


zi ∼ p(zi | δi).

M1

M2







k , δk ∈ S.


zi ∼ p(zi | δi).

M1

M2


Posterior Distribution - Mode Sequence

The posterior probability of a mode sequence {δij}kj=1 is

computed recursively by

wik︸︷︷︸

NewWeight

∝ wik−1︸︷︷︸Old

Weight

·Πδik,δik−1

k︸︷︷︸TransitionProbability

Matrix

· N (yk | yik, S

ik)︸︷︷︸

Likelihood

·p(zk | δik).︸︷︷︸Mode

ObservationPDF

M1

M2





wik︸︷︷︸

NewWeight


Weight

·Πδik,δik−1


Matrix

· N (yk | yik, S

ik)︸︷︷︸

Likelihood


ObservationPDF

M1

M2





wik︸︷︷︸

NewWeight


Weight

·Πδik,δik−1


Matrix

· N (yk | yik, S

ik)︸︷︷︸

Likelihood


ObservationPDF

M1

M2





wik︸︷︷︸

NewWeight


Weight

·Πδik,δik−1


Matrix

· N (yk | yik, S

ik)︸︷︷︸

Likelihood


ObservationPDF

M1

M2





wik︸︷︷︸

NewWeight


Weight

·Πδik,δik−1


Matrix

· N (yk | yik, S

ik)︸︷︷︸

Likelihood


ObservationPDF

M1

M2


Posterior Distribution - State

The posterior distribution of the state is given by

p(xk | Y1:k,Z1:k) =

|S|k∑i=1

wik · N (xk | xi

k, Pik).

M1

M2




p(xk | Y1:k,Z1:k) =

|S|k∑i=1

wik · N (xk | xi

k, Pik).

M1

M2




p(xk | Y1:k,Z1:k) =

|S|k∑i=1

wik · N (xk | xi

k, Pik).

M1

M2




p(xk | Y1:k,Z1:k) =

|S|k∑i=1

wik · N (xk | xi

k, Pik).

M1

M2




p(xk | Y1:k,Z1:k) =

|S|k∑i=1

wik · N (xk | xi

k, Pik).

M1

M2


Bird - ModelThe modes are S = {s, f}.

The state variables are x =

xybs

bf

=

pbs

bf

.

The state-space model is

M1

M2




xybs

bf

=

pbs

bf

.


M1

M2




xybs

bf

=

pbs

bf

.


xk = xk−1 +wk, wk ∼ N (0, Q(δk)),

yk =

(h(pk)

bδkk

)+ vk, vk ∼ N (0, R),

M1

M2




xybs

bf

=

pbs

bf

.


xk = xk−1 +wk, wk ∼ N (0, Q(δk)),

yk =

(h(pk)

bδkk

)+ vk, vk ∼ N (0, R),

M1

M2




xybs

bf

=

pbs

bf

.


xk = xk−1 +wk, wk ∼ N (0, Q(δk)),

yk =

(h(pk)

bδkk

)+ vk, vk ∼ N (0, R),

M1

M2




xybs

bf

=

pbs

bf

.


xk = xk−1 +wk, wk ∼ N (0, Q(δk)),

yk =

(h(pk)

bδkk

)+ vk, vk ∼ N (0, R),

M1

M2




xybs

bf

=

pbs

bf

.


xk = xk−1 +wk, wk ∼ N (0, Q(δk)),

yk =

(h(pk)

bδkk

)+ vk, vk ∼ N (0, R),

M1

M2


Bird - Model ExtensionThe direct mode observation is the radial position

zk =√x2k + y2k,

where p(zk|δk) is given by:

Radius(mm)

0 100 200 300

Pro

babili

ty D

ensity

0

0.2

0.4

0.6

0.8

1

M1

M2


Bird - Model ExtensionThe direct mode observation is the radial position

zk =√x2k + y2k,

where p(zk|δk) is given by:

Radius(mm)

0 100 200 300

Pro

babili

ty D

ensity

0

0.2

0.4

0.6

0.8

1

M1

M2


Bird - Results

Time(s)

0 1000 2000 3000

Mode P

robabili

ty

0

0.5

1

Cage 1

Time(s)

0 1000 2000 3000

Mode P

robabili

ty

0

0.5

1

Cage 4

Time(s)

0 1000 2000 3000

Mode P

robabili

ty

0

0.5

1

Cage 2

Time(s)

0 1000 2000 3000

Mode P

robabili

ty

0

0.5

1

Cage 3

• Estimated modes

• Extracted takeoffs

• Takeoff directions

• Matched takeoffs

M1

M2


Bird - Results

Time(s)

0 1000 2000 3000

Cage 1

Time(s)

0 1000 2000 3000

Cage 4

Time(s)

0 1000 2000 3000

Cage 2

Time(s)

0 1000 2000 3000

Cage 3

• Estimated modes




M1

M2


Bird - Results

Cage 1 Cage 4

Cage 2 Cage 3

• Estimated modes




M1

M2


Bird - Results

Cage 1 Cage 4

Cage 2 Cage 3

• Estimated modes




M1

M2


Bird - Complete Framework M1

M2



Conclusions

Theory is presented on

• a constrained motion model

• an uncertain timestamp model

• direct mode observations

Demonstration through various applications


Conclusions







Conclusions







Conclusions







Future Work

Possible future work is

• multiple observations with uncertain timestamps

• tracking using non-stationary cameras

• motion models tailored to other types of targets

• more advanced tracking algorithms


Future Work







Future Work







Future Work






Thank you for listening!www.liu.se

www.liu.se

Documents

Tracking of Animals Using Airborne Camerasusers.isy.liu.se/en/rt/clave42/presentations/presentation_lic2016.pdf · Licentiate’sPresentation ClasVeibäck November22,2016 3 Introduction