Vision System for Wing Beat Analysis of Bats in the Wild

1Boston University Department of Computer Science2Boston University Department of Biology

Mikhail Breslav1, Nathan W. Fuller2, and Margrit Betke1

ComputerScience

Motivation• Behavior• Aerial Vehicles

Past Work

• Study Kinematics (Hubel. 2012)

– Wind Tunnel– High Resolution Cameras

• 3D Tracking (Wu. 2009)

– Outdoor environment – Model Bats as Points

• Behavior and Trajectory Analysis (Theriault. 2010, Fisher. 2010)

Goal• Estimate Wing Beat Frequencies– Potential to improve tracking

TWing Beat Frequency: 1/T

Hubel et al. 2012

Challenging Data• Unpredictable Motions• Relatively Low Resolution• In FOV for a Short Time

Segmentation and Tracking• Foreground / Background Estimation• Kalman Filter

Shape-time Signals• Output of Tracker• Define “Shape”

Prototype Shapes

• Assumption– There are shapes that uniquely identify 3D

poses for a given camera

• Example

• Currently chosen manually

“up” “down” “neutral”

Main Idea• A prototype shape is equal to a 3D

pose• Repeating prototype shapes in a

shape-time signal

Estimate Wing Beat

Shape Comparison• Shape Distance– Shape Context Descriptor (Belongie et al. 2002)• Invariant to translation, scale, and optionally rotation

– Hungarian Algorithm• Establish Correspondences

• Estimate Wing Relative to Body with feature W

Shape Similarity Scores• Use Shape Distance and Ratio W to assign

similarity score– Also consider the ‘none’ hypothesis

‘None’

.24 .16 .43 .17

“up” “down” “neutral”

Process Shape-Time Signal

• Find confident matches to prototype shapes

“up”

Process Shape-Time Signal

“down”

“neutral”

Time Axis

Fast Fourier Transform

Time Axis

“up”

“down”

“neutral”

FFT

Fast Fourier Transform

Periodicity Estimate of 9.76 Hz

Experimental Results• 20 Bats • Both Automatic and Manual estimates

Discussion• Reasonable Estimates– Deviates from manual annotations by 1.3 Hz on

average, standard deviation 1.8 Hz– Falls within 10-15 Hz as reported in biology

literature (Foehring. 1984)

• Main Contribution– System for using shapes to estimate wing beat– First to do this for bats in the wild • Vision based system

Future work

• Choosing prototype shapes – Automatically and Intelligently

• Understand the mapping between 2D shapes and 3D poses for a given model– Generalize across datasets

• Try more robust shape comparison measures

Thank You for your Attention!

Questions?

Holding a bat!