Dynamic Scenes by Image Sequence Analysis

1

Dynamic ScenesDynamic Scenes

by by

Image Sequence AnalysisImage Sequence Analysis

Dynamic ScenesDynamic Scenes

by by

Image Sequence AnalysisImage Sequence Analysis

Jun ShenJun Shen2004

2

Presentation schemePresentation scheme General presentation Dynamic scene analysis (DSA): a general view Motion detection

by background subtraction & by orthogonal moments 3D-model-based vehicle pose determination &

tracking Face tracking Gait tracking (Model-based tracking) Automatic gait recognition Learning & recognition of activity patterns by

fuzzy self-organizing Kohonen net Demonstration of results

3

I. General I. General PresentationPresentation

4

General framework of visual surveillance. . .

Fusion of Information from multiple cameras

Camera 1

Environment modeling

Motion segmentation

Object classification

Tracking

Behavior understanding and description

Personal identification

Camera n

PROCESSING

PROCESSING

I. General presentation

5

II. Dynamic Scene II. Dynamic Scene

Analysis (DSA):Analysis (DSA):

A general viewA general view

6

II. Dynamic Scene Analysis (DSA):II. Dynamic Scene Analysis (DSA):A general viewA general view

Low-level analysis

Motion detection Pose determination Hidden effect processing Moving object classification Tracking

II. DSA: A general view

7

Motion detection methods– Background subtraction

– Temporal difference between successive frames

– Optic flow

– Matching: correlation, etc

– Frequency domain methods

Pose determinationII. DSA: A general view

8

Moving object classificationMoving object classification

Classification based on geometric and radiometric properties of object– Shape– Size– Height-width ratio– Color– Texture– Features

Classification based on motion


9

TrackingTracking

Tracking based on regions

Tracking based on moving contours

Tracking based on features

Tracking based on object models


10

Behavior understanding and Behavior understanding and descriptiondescription

Finite state automate

Non-deterministic automate

Hidden Markov process model

Neural nets

Syntactic methods

….


11

Person identification by gait analysis Person identification by gait analysis for video surveillancefor video surveillance

Model-based methods Statistical methods Characteristic-parameter-based methods Temporal-Spatial-motion-based methods Combination of gait analysis with other

biometrics methods


12

Fusion of information from Fusion of information from multiple camerasmultiple cameras

Positioning of cameras Calibration of cameras Matching of objects from multi-camera Switching of cameras Fusion of information from multi-camera Hidden effect processing using multiple

cameras


13

III. Motion detectionIII. Motion detection

14

III. Motion detectionIII. Motion detection

Background subtraction Temporal Gaussian-Hermite moments

– Moments

– Orthogonal moments

– Gaussian-Hermite moments

– Motion detection by Gaussian-Hermite moments

III. Motion detection: 1. Background subtraction

15

III.1. III.1. Motion detection Motion detection in color (or gray value) in color (or gray value)

image sequence image sequence by by

background subtractionbackground subtraction

16

System overview

Based on background subtraction

FilteringBackground

image creation

Moving pixel

detection

Illumination change

elimination

Shadow eliminationLabeling

Input imagesequence

Moving objectsdetected


17

Filtering & background image (c'tnd)A mobile object stops during a period > half the temporal W. size, It would be considered as static object and backgr'd updating will take moving object color.When it begins to move again, backgr'd image thus updated would disturb the detection of its motion (double moving objects

detected).

False moving object


18

Solution –Color of moving pixels

not taken into account

in backgr'd updating.

–Distinguishing stopped

“mobile” objects from

real static objects.

–Comparison of present

& preceding positions

tells in motion or a

stopped mobile object.

Filtering & background image (c'tnd)


19

Motion detection by background subtraction for color images

Difference between current frame & backgr'd

Current image

Background imageDiff

Difference imageR,B,G

Channels of Difference

Image


20

Motion detection by b'gd subtraction (c'tnd)

Difference between current frame & backgr'd Segmentation of the difference color image

– Fuzzy segmentation of R,B,G channels separatelyAutomatic determination of threshold T,Fuzzy set “mobile pixels” by non-sym. m'ship function.

– Fuzzy segmentation with 3 channels together


21



– Fuzzy segmentation of R,B,G channels separatelyAutomatic determination of threshold T

T

hi

i

Threshold by "Max. Distance"

Immobile Fuzzy set of mobile


22


– Fuzzy segmentation of R,B,G channels separately Automatic determination of threshold T, Fuzzy set “mobile pixels” by non-sym. m'ship

function.

– Fuzzy segmentation with 3 channels together

T c= dmax+ k/( dmax- dmin), (k>0)

dmax and dmin, max. & min. intensities.

+

(x)

x



23

ATD (Automatic Threshold by max. distance)

Actual color frame Background image

Fuzzy M’ship fn

Automatic threshold by moment conservation method

Difference Image

ATD

B channel G channel ATD

Fuzzy M’ship fnFuzzy M’ship fn

Fuzzy deduction

Mobile pixel image

R channel

Fuzzy Segmentation of Difference ImageIII. Motion detection: 1. Background subtraction

24

Elimination of false motion due to illumination change

Problem Bg'd image update using

preceding frames not fast adapted to illumination v.

Rapidness of bg'd adaptation depends on temporal window size & bg'd adaptation method.

Even auto-adaptation used, bg'd adapted to illumination change only after an accumulation of frames


25

AND

Validated mobile pixels

Mobile pixels detected by background subtraction for

the current frame

Mobile pixels detected in

preceding frame

Mobile pixels detected by variation

in successive frames

OR

Diagram of false motion elimination


26

Shadow Elimination Problem: Shadows of moving objects being of

almost the same motion as moving objects Importance of shadow elimination

Obtaining more precise description of moving objects

Center of gravity


27

III.2. Motion detection by III.2. Motion detection by

orthogonal momentsorthogonal moments

28

III.2. Motion detection by III.2. Motion detection by orthogonal moments orthogonal moments

III. Motion detection: 2. G-H moments

Moments

– Geometric, Legendre & Hermite moments

– Behavior in space & frequency domains

– Gaussian-Hermite (G-H) moments Motion detection by G-H moments Comparison with other methods Concluding remarks

29

Geometric, Legendre & Hermite Geometric, Legendre & Hermite moments and their calculationmoments and their calculation

Geometric moments and their calculation 1D geometric moments Mn(x) at point x: Mn(x)= S(x+ t) tn dt n= 0, 1, 2, ...

2D geometric moments of a 2D image I(x, y):– Mm, n(x, y)= I(x+ u, y+ v) um vn du dv

Fast algorithms, such as Pascal Triangle. Explicit statistical signification. Functional analysis viewpoint: Signal projected onto

polynom. space, taking monomial functions as bases.

w

w

1

1

w

w

2

2

w

w


30

Orthogonal Legendre moments

Using orthogonal bases:– Calculation could be reduced, – Error easier to estimate when limited proj. used, – Reconstruction simpler.

Orthogonal Legendre polynomials: (dn/ dxn) (x2- 1)n / (2n. n!) for x [-1, 1],

Pn(x) = {

0 otherwise.


31

Scaled Legendre polynomials: [(dn/ dxn) (x2- w2)n ]/ [(2 w)n. n!] for x [-w, w]

Ln(x) = { 0 otherwise.

n-th order orthogonal Legendre moment:

Mn(x) = S(x+ t) Ln(t) dt

= <L0(t), S(x+ t)> (inner product).

w

w


32

Recursive calculation of Legendre moments

The nth order orthogonal L. moments, calculated from window [x- w, x+ w], can be computed from (n- 1)th & (n- 2)th order L.M. : M0(x) = <L0(t), S(x+ t)> = S1(x+ w) - S1(x- w)

M1(x) = <L1(t), S(x+ t)> = [S1(x+ w) + S1(x- w)] - <L0(t), S1(x+ t)> / w

Mn(x) = <Ln(t), S(x+ t)> = <Ln-2(t), S(x+ t)> - [(2n- 1)/ w] <Ln-1(t), S1(x+ t)> , for n> 1with S0(t)= S(t) and Si(t)= Si-1(y) dy for i= 1, 2, 3, …

Si(t) easily calculated from Si-1(t) by recursive sum-box tech.

t


33

2D Legendre moments

In 2D cases:

kx ky

Mp, q(x, y)= I(x+ t, y+ v) Lp(t) Lq(v) dt dv -kx -ky

Separable, decomposed into cascade of 1D calculation, by recursive algo.


34

Hermite moments Scaled Hermite polynomial

Pn(t)= Hn(t/)with Hn(t)= (-1)n exp (t2) (dn/ dtn) exp (-t2).

1D n-th order Hermite moment:Mn(x, S(x))= Pn(t) S(x+ t) dt n= 0, 1, ...

2D Hermite moments of an image I(x, y):Mp,q(x,y,I(x,y))= Hp,q(t/, v/) I(x+t, y+v) dt dv

with Hp,q (t/, v/)= Hp(t/) Hq(v/). Separable, calculated by cascade of 1D.


35

Behavior of geometric, Legendre & Hermite

Moments in space & frequency

domains Importance of behavior analysis

Behavior in space domain

Behavior in frequency domain


36

Geometric moment base functions– Graphs of similar shapes, – Moments considered as projections onto base function

space, not efficient for diff.spatial modes. Hermite & Legendre mnt. base functions

– Many oscillations, depending on the order,– Extract efficiently characteristics of diff. spatial modes

(orthogonal polynomial of order n has n diff. zero-crossings).

Oscillations in Hermite bases much less important than Legendre ones (because the Hermite bases are not really orthogonal).

Same conclusion holds in 2D cases.


37

Geometric moment base functions:– low-pass kernel, FT monotonically decreased.

Hermite moment base functions:– as order increased, max. FT position moves to right,

and more and more similar to a band-pass kernel. Legendre moment base functions:

– best band-pass characteristics except for very low orders. The higher the order is, the more to the right the pass-band moves.

L. moments separate characteristics in different frequency bands better than H. moments, which are in turn better than geometric ones.


38

Gaussian-Hermite Moments Problem:Discontinuity in Geometric, H. & L. moment base at window boundary.How better represent local characteristics of (noisy) images? Solution: Smoothed orthogonal Gaussian-Hermite (G-H) momentsOrthogonal moments with Gaussian smoothing window function

Gaussian smoothing function g(x, )= (2 2) -1/2 exp (-x2/ 22) n-th order smoothed G-H moment:

Mn(x, S(x))=

Bn(t) S(x+ t) dt n= 0, 1, ...

with Bn(t)= g(t, ) Pn(t)Pn(t): scaled Hermite polynomial function of order n defined by

Pn(t)= Hn(t /)

with Hn(t)= (-1)n exp (t2) (dn/ dtn) exp (-t2)


39

Property of G-H moments

Orthogonal moments.

Recursively calculated as follows: Mn(x, S(m)(x))= 2(n-1) Mn-2(x, S(m)(x)) + 2Mn-1(x, S(m+1)(x))

for m>=0 and n>= 2

with M0(x, S(m)(x))= g(x, ) * S(m)(x)) for m>= 0

M1(x, S(m)(x))= 2 d[g(x, )]/ dx * S(m)(x)) for m>= 0

and in particular,

M0(x, S(x)) = g(x, ) * S(x)

M1(x, S(x)) = 2 d[g(x, ) * S(x) ] / dx

where S(m)(x) dmS(x) / dx

m, S(0)(x) S(x).


40

Comparison G-H moments better separate diff.

bands. Larger quality factor

Q= (Center freqency)/ (Effective bandwidth). G-H moments & G.-filtered deriv.:

– G-H moments: linear combinations of Gauss-filtered derivatives of signal.

– Construct orthogonal features from Gaussian-filtered derivatives.


41

G.-H. moments & wavelet analysis Derivatives of Gaussians widely used as mother

wavelets, Different order derivatives of Gaussian filters

define different wavelets, Derivatives filtered by Gaussian filters of different

represent the decomposition of signal into wavelets.

Smoothed orthogonal Gaussian-Hermite moments offer a solution to construct orthogonal features from the wavelet analysis results.


42

2D orthogonal G-H moments

Mp, q(x, y, I(x, y)) =

G(t, v, ) Hp,q (t/, v/) I(x+ t, y+ v) dt dv

with Hp,q (t/, v/)= Hp(t/) Hq (v/),

scaled 2D Hermite polynomial.

Separable, cascade of 1D recursive algorithm.


43

Performance comparison: Sensibility to noise

Noise-free images and noisy ones with additional random noise,

Moment vectors (m0,0, m0,1, …, m0,5, m1,0, m1,1, …, m1,5),

Normalized distances between noise-free images and noisy ones.


44

Orthogonality equivalence

To better understand the good performance of orthogonal moments in both spatial and frequency domains, we have

Orthogonality equivalence theorem

- Orthogonal moment base functions are not only orthogonal in spatial domain but also in frequency domain.


45

Experimental verification Three different reference shape images: quadrilateral, hexagon and octagon. Noisy images: adding random noises of diff. standard deviations. Each shape characterized by 12 moments of orders (0,0), ..., (0,5), (1,0), ..., (1,5). Geometric, H. and L. moments are tested. Classification by comparing moment vector of noisy shape with the 3 ref. shapes.


46

Motion detection Motion detection by by Gaussian-Hermite Gaussian-Hermite

momentsmoments Why using G-H moments Motion detectiMotion detectionon using using G-H momentsG-H moments ResultsResults and comparison

– Comparison with differential methodsComparison with differential methods

– Comparison with Comparison with background background subtractionsubtraction

– Comparison withComparison with adaptive background

subtraction


47

Why using G-H moments?– Methods of motion detection in image sequence

Background-subtraction-based, including stochastic estimation of activation

Difficulty– Frame-to-frame illumination changes,

– Slowly moving and/or uninterested moving objects

– Calculation of adaptive background images demanding accumulation of a large number of images.

Based on temporal variation in successive images

Difficulty– Sensibility to noise


48

Advantages of using orthogonal G-H moments for motion detection– G-H moments: linear combinations of image

derivatives, permitting to detect image changes– Much smoother than other moments, therefore much

less sensitive to noises, facilitate moving object detection in noisy image sequences.

– Odd-order G-H moment base functions: linear combinations of odd order derivatives of Gaussian functions.

– Temporal G-H moments: composed of temporal image derivatives to detect moving objects in image sequences.


49

DetectiDetectingng moving target moving targets using s using G-H G-H moments of differenmoments of differentt orders orders

Given an image sequence Calculation of temporal G-H moments M1, M3 and M5

Fuzzy motion detection by moment image segmentation, using threshold by improved invariable-moment--method, using non-sym. Mship function for each point in moment images.

Membership function update by fuzzy relaxation: spatial relation between pixels in single and successive frames

Moving pixel decision


50

Comparison with differential methodsComparison with differential methods

Comparison with other methods


51

Comparison with Comparison with background background subtractionsubtraction

Test image sequence:illumination changed in some frames

Background subtraction method fails for illum. changed frames

G-H moments succeed


52

Comparison withComparison with adaptive adaptive background subtractionbackground subtraction

Adaptive back’d subtr’n improving motion

detection

Problem: back’d updating para. value choice,

depending on motion velocities

G-H moments: problems much better solved.


53

Example: an image sequenceExample: an image sequence


54

Motion detection result by G-H momentMotion detection result by G-H moment


55

Moving car trajectory (Moving car trajectory (Spline)Spline)


56

IV. 3D-model-based IV. 3D-model-based

Pose determination &Pose determination &

Tracking of vehicles Tracking of vehicles

IV. Pose and tracking

57

System configurationSystem configuration

Camera model

Image sequence

3D vehicle model

Low-level video tracking

High-levelbehavior analysis


58

System frameworkSystem framework

New target

s?

Motion detection

Initialization

3D pose estimation

Model projection

Pose optimization Pose quality evaluation

Obstacle hiding analysis

NN

YY

Behavior analysis and semantic description

Image sequence Camera calibration Modeling

Low-level processing

High-level processing


59

Vehicle pose determinationProblems:

Detection of region of interest containing

a vehicle

Motion detection

Classification of moving objects

Determination of 3D pose of the vehicle


60

Known dataKnown data

ROI containing the vehicle on the image

3D model of the vehicle

Camera intrinsic and extrinsic parameters

Road surface plane constraint

Initial vehicle pose estimation


61

Pose quality evaluationPose quality evaluation

Model features:

Selected straight edge segments of the 3D model

Image features:

Edge points detected on the image

Quality based on PLS (Point to Line Segment) distance


62

Pose optimizationPose optimizationMake move 3D model in 3D space, from the

initial pose estimation to optimal pose3D model moves on the road surface plane

(Plan motion constraint):Translation on the road planeRotation around axis normal to road plane & passing through vehicle’s center of gravity

Weak projective projection hypothesis:Translation and rotation above independent on the image plane

DecompositionTranslation optimization + Rotation optimization


63

Translation optimizationTranslation optimization

For the projection Lp of the pth segment of 3D

model, define a subset of I:

Pose error function2

),(,,, )),(]/x([),(

p Ikj

pkjpkjpkj

p

LIDngngLIH

),(min),(| ,,, nkjn

pkjkjp LIDLIDIII


64

Determination of rotationDetermination of rotation

ZM

YMXM

OM

ZW

YWXW

OW

ZC

YC

XC OC

M: Reference system on the model objectW: Reference system in 3D worldC: Reference system on the camera


65

Rotation optimizationRotation optimization

After translation optimization

Searching in a small interval centered at

the estimated rotation angle

Take the angle that minimizes the pose error function


66

Hidden effect detection and Hidden effect detection and visible region determinationvisible region determination

Different types of hidden effectCase 1: Moving object hidden by backgroundCase 2: Moving object leaving or entering in

the vision fieldCase 3: Moving object hidden by other moving

objects Hidden effect detection Visible region determination


67

Experimental resultsExperimental results


68

Principle of TrackingPrinciple of Tracking Non-deformable solid object tracking

(Vehicles, …)

For the entire object:– Shape, size, color, ...– Estimation from motion

Solid objects with joints (Human body, etc)

For each part of the object: – Sub-model: shape, size, color, ...– Estimation from the motion


69

V. Face TrackingV. Face Tracking

70Overview of AlgorithmOverview of Algorithm

An

inp

ut

seq

uen

ce

Template Confidence Measure > Threshold

Y

Motion Detection

Set Search Region

Body-part Constraints

NFace

Detection

The first frame in an image sequence

Template Initialization

Template Matching

Template Updating

H Profile

V Profile

V. Face tracking

71

VI. Gait TrackingVI. Gait Tracking

(Model-based tracking)(Model-based tracking)

72

Gait TrackingGait Tracking(Model-based tracking)(Model-based tracking)

Pose optimization

Human body model

Current frame

Human body pose in the preceding

frame or

the initial pose

Pose estimation

Motion Model

Motion constraints

Tracking result

Application

Initialization Dynamic modelMotion model

Motion constraints

Search strategyHuman body model

Pose evaluation function

Motion synthesisGait recognition

VI. Gait tracking

73

Model representation & LearningModel representation & Learning

- Geometric model of human body- Geometric model of human body

• Generalized cylinder model

• Motion parameters: 1021 ,,,,, yxP

VI. Gait tracking

74

VII. Automatic Gait VII. Automatic Gait

RecognitionRecognition

75

VII. Automatic Gait RecognitionVII. Automatic Gait Recognition

Gait: – Useful biometric feature for recognition – Attractive modality of human identification at

a great distance, for surveillanceApplication: automated person identification for

surveillance or monitoring systems in security-sensitive environments such as banks, parking lots and military bases.

Method based on Statistical Shape Analysis

VII. Gait recognition

76

Advantages– The only perceivable biometric at a distance;– Not requiring proximal contact; – Easy to capture;– Difficult to conceal.

Disadvantages– A large amount of data;– Intermediate recognition accuracy;– Subject to some physical conditions such as drunkenness, pregnancy, and injuries involving joints.

Advantages and Disadvantages


77

Camera

Gait image Sequence

Tracking

Background image

creation

Motion Detection

Gait Feature

Extraction

Classifier Database

Recognition Results

Monitoring Area

General framework of gait recognition


78

VIII. VIII. Learning & Recognition Learning & Recognition of Patterns of Activity of Patterns of Activity

by by Fuzzy Self-Organizing Fuzzy Self-Organizing

Kohonen NetworkKohonen Network

79

VIII. Learning & Recognition of VIII. Learning & Recognition of Patterns of Activity by Fuzzy Self-Patterns of Activity by Fuzzy Self-Organizing Kohonen NetworkOrganizing Kohonen Network• Activity understandingActivity understanding

in particular,in particular,

• Learning of activity patterns Learning of activity patterns

• Anomaly detectionAnomaly detection

• Activity predictionActivity prediction

VIII. Activity patterns

80

General SchemaGeneral Schema

1. Moving target tracking

2. Trajectory coding

3. Activity recognition– Data acquisition

– Recognition structure

– Learning

– Recognition


81

•Recognition structure • Why Self-Organizing Kohonen Network?• Classical activity recognition systems?Classical activity recognition systems?

• Depending on predefined activity patternsDepending on predefined activity patterns• Non adaptable to changing environmentsNon adaptable to changing environments

Highly desirable to establish general approach Highly desirable to establish general approach of of activity recognition able to automatically activity recognition able to automatically generate activity modelsgenerate activity models..

Kohonen self-organizing topological mapKohonen self-organizing topological map‘‘Winner takes all’Winner takes all’

Using Using Fuzzy Fuzzy self-organizing Kohonen netself-organizing Kohonen net


82

Learning

• Training data: Training data: Set of training trajectories


83

Anomaly DetectionAnomaly Detection

• Detecting abnormal trajectoryDetecting abnormal trajectoryGiven a trajectory:

• We first look for the neuron that best matches it, which gives the class to which it is classified.

• If the Euclidean distance between the input trajectory code and the best matched neuron is greater than a threshold q, the activity represented by the trajectory is considered as unusual (unusual (abnormal).


84

Prediction of ActivityPrediction of Activity Given a part of a motion trajectory:Given a part of a motion trajectory:• Sampling this part to get a "sub-sample" vector

V.• Mismatching score between the sub-sample and

each neuron i by the Euclidean distance • The probability of each possible future motion

trajectory along which the object• According to the probabilities thus determined,

several future trajectories can be predicted with

probability. VIII. Activity patterns

85

IX. IX. Demonstrations Demonstrations

of resultsof results

86

Thank you!Address:

Jun ShenInstitut EGID - Bordeaux 31, Allée Daguin33607 Pessac cedexFRANCE

Email: [email protected]

Phone: (+33) 5 57 12 10 26Fax: (+33) 5 57 12 10 01

Documents

Dynamic Scenes by Image Sequence Analysis