Tracking of Animals Using Airborne...

Tracking of AnimalsUsing Airborne CamerasClas Veibäck

1 Introduction2 Target Tracking3 Camera Sensor4 Constrained Motion Model5 Uncertain Timestamp Model6 Mode Observations7 Conclusions

Licentiate’s Presentation Clas Veibäck November 22, 2016 3

Introduction

• Tracking of animals

• Overhead cameras

• Contributions

• Applications

Introduction

• Contributions

• Applications

Introduction

• Contributions

• Applications

Introduction

• Contributions

• Applications

Main Contributions

Uncertain Timestamps

Constrained Motion Model Mode Observations

k− 1 k k+ 1

Main Contributions

Constrained Motion Model

Mode Observations

k− 1 k k+ 1

Main Contributions

Constrained Motion Model Mode Observations

k− 1 k k+ 1

Dolphin Application

• Dolphinarium at Kolmården

Wildlife Park

• Fisheye camera with

occlusions

• Reflections and changing

light conditions

• Constrained to basin

Dolphin Application

Wildlife Park

occlusions

light conditions

Dolphin Application

Wildlife Park

occlusions

light conditions

Dolphin Application

Wildlife Park

occlusions

light conditions

Bird Application

• Recording of birds in Emlen

funnels

• Detect take-off times

• Estimate take-off directions

• Stationary and flight modes

Bird Application

funnels

Bird Application

funnels

Bird Application

funnels

Orienteering Application

• GPS trajectory

• Control position known

• Improve position estimate

• GPS trajectory

Target Tracking

Tracker

Presentation

Pre-processing Association TrackManagement

Estimation

Sensor Data

Pre-processing

• Raw Sensor Data

• Signal and Image Processing

• Detections

Tracker

Presentation

Estimation

Sensor Data

Pre-processing

• Raw Sensor Data

• Detections

Tracker

Presentation

Estimation

Sensor Data

Pre-processing

• Raw Sensor Data

• Detections

Tracker

Presentation

Estimation

Sensor Data

Association

• Targets

• Detections

• False and Missed Detections

• Tracks

• Hypothesis

• Multiple Hypotheses

Tracker

Presentation

Estimation

Sensor Data

Association

• Targets

• Detections

• Tracks

• Hypothesis

Tracker

Presentation

Estimation

Sensor Data

Association

• Targets

• Detections

• Tracks

• Hypothesis

Tracker

Presentation

Estimation

Sensor Data

Association

• Targets

• Detections

• Tracks

• Hypothesis

Tracker

Presentation

Estimation

Sensor Data

Association

• Targets

• Detections

• Tracks

• Hypothesis

Tracker

Presentation

Estimation

Sensor Data

Association

• Targets

• Detections

• Tracks

• Hypothesis

Tracker

Presentation

Estimation

Sensor Data

Track Management

• New track

• Confirmed track

• Dead track

Tracker

Presentation

Estimation

Sensor Data

Track Management

• New track

• Confirmed track

• Dead track

Tracker

Presentation

Estimation

Sensor Data

Track Management

• New track

• Confirmed track

• Dead track

Tracker

Presentation

Estimation

Sensor Data

Estimation

• Given association hypothesis

• Properties of each target

• Over time using model

Tracker

Presentation

Estimation

Sensor Data

Estimation

Tracker

Presentation

Estimation

Sensor Data

Estimation

Tracker

Presentation

Estimation

Sensor Data

Estimation

Linear Gaussian state-space model

xk = Fkxk−1 +wk, wk ∼ N (0,Qk),

yj = Hjxj + vj , vj ∼ N (0,Rj),

x0∼ N (x0,P0).

Posterior distribution

p(xk|x0,y1, . . . ,yk)

Tracker

Presentation

Estimation

Sensor Data

Estimation

x0∼ N (x0,P0).

p(xk|x0,y1, . . . ,yk)

Tracker

Presentation

Estimation

Sensor Data

Estimation

x0∼ N (x0,P0).

p(xk|x0,y1, . . . ,yk)

Tracker

Presentation

Estimation

Sensor Data

Estimation

x0∼ N (x0,P0).

p(xk|x0,y1, . . . ,yk)

Tracker

Presentation

Estimation

Sensor Data

Estimation

x0∼ N (x0,P0).

p(xk|x0,y1, . . . ,yk)

Tracker

Presentation

Estimation

Sensor Data

Estimation

x0∼ N (x0,P0).

p(xk|x0,y1, . . . ,yk)

Tracker

Presentation

Estimation

Sensor Data

Estimation

x0∼ N (x0,P0).

p(xk|x0,y1, . . . ,yk)

Tracker

Presentation

Estimation

Sensor Data

Camera Sensor

Camera Model

• Camera extrinsics

• Camera intrinsics

• Projection

• Perspective compensation

• Lens distortion

Camera Model

• Projection

• Lens distortion

Camera Model

• Projection

• Lens distortion

Camera Model

• Projection

• Lens distortion

Camera Model

• Projection

• Lens distortion

Dolphin - Camera Model z

Bird - Camera Model z

Foreground Segmentation

• Gaussian-mixture

background model

• Handles changing light

conditions

• Handles reflections

• Degree of confidence

background model

conditions

background model

conditions

background model

conditions

• Targets constrained to

region

• Feasible predictions

• Similar behaviour

region

Turning Model

ω(x) = dr(x)

(βd + βa

(p⊥ · l(n)

))w(x, n) dn

• Nearly constant speed

• Influence by edges

• Avoid collision with edges

• Align with edges

• Integrate along edge

• Preferred direction

• Polygon region

Turning Model

ω(x) = dr(x)

(βd + βa

(p⊥ · l(n)

))w(x, n) dn

• Polygon region

Turning Model

ω(x) = dr(x)

(βd + βa

(p⊥ · l(n)

))w(x, n) dn

w(x, n) =1

‖p− n‖2

• Polygon region

Turning Model

ω(x) = dr(x)

(βd + βa

(p⊥ · l(n)

))w(x, n) dn

• Polygon region

Turning Model

ω(x) = dr(x)

(βd + βa

(p⊥ · l(n)

))w(x, n) dn

• Polygon region

Turning Model

ω(x) = dr(x)

(βd + βa

(p⊥ · l(n)

))w(x, n) dn

• Polygon region

Turning Model

ω(x) = dr(x)

(βd + βa

(p⊥ · l(n)

))w(x, n) dn

Clockwise

Counterclockwise

• Polygon region

Turning Model

ω(x) = dr(x)

N∑i=1

(βd + βa(p⊥ · li)

)wi(x)

• Polygon region

Potential Field

Dolphin - Model Simulation

Dolphin - Model Comparisons

Detection regionConstraint regionConstant Velocity modelCoordinated Turn modelConstrained Motion model

Dolphin - Complete Framework

Uncertain Timestamp Model

• Traditional measurements

• Observations sampled at an

uncertain time

• Crime scene investigations

• Traces from animals

uncertain time

Consider a linear Gaussian state space model,

yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

x0 ∼ N (x0,P0),

Extend the model with

yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

x0 ∼ N (x0,P0),

z = Hzxτ + vz, vz ∼ N (0,Rz), τ ∼ p(τ).

yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

x0 ∼ N (x0,P0),

yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

x0 ∼ N (x0,P0),

yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

x0 ∼ N (x0,P0),

yj = Hyjxj + vy

j , vyj ∼ N (0,Ry

x0 ∼ N (x0,P0),

Simple Uncertain Time Scenario

Consider the simple model,

xk = xk−1 + wk, wk ∼ N (0, Q),

yj = xj + vyj , vyj ∼ N (0, Ry)

for k ∈ {1, . . . , N} and two measurements y1 and yN .

z = xτ + vz, vz ∼ N (0, Rz), τ ∼ p(τ).

Consider the simple model,

xk = xk−1 + wk, wk ∼ N (0, Q),

yj = xj + vyj , vyj ∼ N (0, Ry)

for k ∈ {1, . . . , N} and two measurements y1 and yN .

z = xτ + vz, vz ∼ N (0, Rz), τ ∼ p(τ).

0 0.2 0.4 0.6 0.8 1

Positio

z (first case)z (second case)

The measurements are

• y1 = 0,

• yN = 1.

The two cases of

observations are

• z = 0.5 with flat prior.

• z = 1.5 with non-flatprior.

0 0.2 0.4 0.6 0.8 1

Positio

• y1 = 0,

• yN = 1.

The two cases of

observations are

0 0.2 0.4 0.6 0.8 1

Positio

• y1 = 0,

• yN = 1.

The two cases of

observations are

Posterior Distributions - Time

The posterior distribution of the uncertain time is

wτ , p(τ |Y, z)︸︷︷︸Posterior

∝ p(τ)︸︷︷︸Prior

N (z| zτ , Sτ )︸︷︷︸Likelihood

p(τ |Y, z)maxX p(X , τ |Y, z)p(τ)

0 0.5 1

babili

×10-3

0 0.2 0.4 0.6 0.8 1

×10-3

Posterior Distributions - States

The posterior distribution of the state is

p(xk|Y, z) =

N∑τ=1

wτ · N (xk|xτk,P

p(xk|Y, z) =

N∑τ=1

wτ · N (xk|xτk,P

p(xk|Y, z) =

N∑τ=1

wτ · N (xk|xτk,P

p(xk|Y, z) =

N∑τ=1

wτ · N (xk|xτk,P

0 0.2 0.4 0.6 0.8 1

Estimators - Minimum Mean Squared Error

XMMSE |Y, z

XMMSE |Y

0 0.5 1

Positio

0 0.5 1

Positio

Orienteering - Results

• Adjusted trajectory

• Single control point

Orienteering - Results

• Adjusted trajectory

• Single control point

Mode Observations

k− 1 k k+ 1

Mode Observations

k− 1 k k+ 1

• Jump Markov model

• Model target behaviour

• Probability of switching

• Direct mode observation

Mode Observations

k− 1 k k+ 1

Mode Observations

k− 1 k k+ 1

Mode Observations

k− 1 k k+ 1

Jump Markov ModelThe jump Markov linear model is

xk = Fk(δk)xk−1 +wk, wk ∼ N (0, Qk(δk)),

yj = Hj(δj)xj + vj , vj ∼ N (0, Rj(δj)).

The mode is modelled as

p(δk | δk−1) = Πδk,δk−1

k , δk ∈ S.

Augmented is the direct mode observation,

zi ∼ p(zi | δi).

k , δk ∈ S.

zi ∼ p(zi | δi).

k , δk ∈ S.

zi ∼ p(zi | δi).

k , δk ∈ S.

zi ∼ p(zi | δi).

Posterior Distribution - Mode Sequence

The posterior probability of a mode sequence {δij}kj=1 is

computed recursively by

wik︸︷︷︸

NewWeight

∝ wik−1︸︷︷︸Old

Weight

·Πδik,δik−1

k︸︷︷︸TransitionProbability

Matrix

· N (yk | yik, S

ik)︸︷︷︸

Likelihood

·p(zk | δik).︸︷︷︸Mode

ObservationPDF

wik︸︷︷︸

NewWeight

Weight

·Πδik,δik−1

Matrix

· N (yk | yik, S

ik)︸︷︷︸

Likelihood

ObservationPDF

wik︸︷︷︸

NewWeight

Weight

·Πδik,δik−1

Matrix

· N (yk | yik, S

ik)︸︷︷︸

Likelihood

ObservationPDF

wik︸︷︷︸

NewWeight

Weight

·Πδik,δik−1

Matrix

· N (yk | yik, S

ik)︸︷︷︸

Likelihood

ObservationPDF

wik︸︷︷︸

NewWeight

Weight

·Πδik,δik−1

Matrix

· N (yk | yik, S

ik)︸︷︷︸

Likelihood

ObservationPDF

Posterior Distribution - State

The posterior distribution of the state is given by

p(xk | Y1:k,Z1:k) =

|S|k∑i=1

wik · N (xk | xi

k, Pik).

p(xk | Y1:k,Z1:k) =

|S|k∑i=1

wik · N (xk | xi

k, Pik).

p(xk | Y1:k,Z1:k) =

|S|k∑i=1

wik · N (xk | xi

k, Pik).

p(xk | Y1:k,Z1:k) =

|S|k∑i=1

wik · N (xk | xi

k, Pik).

p(xk | Y1:k,Z1:k) =

|S|k∑i=1

wik · N (xk | xi

k, Pik).

Bird - ModelThe modes are S = {s, f}.

The state variables are x =

The state-space model is