Human Activity AnalysisA Review

Volume 43Issue 3,April 2011.ACM New York, NY, USA

Presented by:Sonam yar

J.K .Aggarwal and M.S.RyooThe University of Texas at Austin

CONTENTS

Problem Domain

Human activities

Introduction

Single layer approaches

Hierarchical Approaches

Human-object Interactions

Group Activities

Conclusion

PROBLEM DOMAIN

The increased use of cameras

The most important: goals of video analytics is to detect abnormalities

HUMAN ACTIVITIES

Gestures

Actions

Interactions

Group Activities

Applications of Human Activity analysis

Automated surveillance systemsAirportsSubway stationsPatients observanceAnalysis of the physical condition of people Caring of aged people

INTRODUCTION

– Human activity recognition is important.

– Objective of the paper

– Overview

– Concentrates on low level along with high level activity recognition methodologies

– Approach based taxonomy

TAXONOMY OF ACTIVITIES

Automated surveillance systemsAirportsSubway stationsPatients observanceAnalysis of the physical condition of people caring of aged people

SINGLE LAYER APPROACHES

1. Action recognition with space-time volumes– Bobick and Davis template matching

• Motion-history image ( MHI)• Motion-energy image ( MEI)

Space-time Approaches

SINGLE LAYER APPROACHES

Space-time Approaches

SINGLE LAYER APPROACHES

•Continue….

oShechtman and Irani Compare volumes in terms of their patchesoKe et al. Used segmented spatio-temporal volumes to model human activities.

Space-time Approaches

SINGLE LAYER APPROACHES

Continue….oRodriguez et al

Filters capturing characteristics of volumes

Space-time Approaches

SINGLE LAYER APPROACHES

Continue….o Rodriguez et al

Filters capturing characteristics of volumes

SINGLE LAYER APPROACHES

•Disadvantages of space-time volumeThe major disadvantage of space-time volume approaches is the difficulty in recognizing actions when multiple persons are present in the scene.

Space-time Approaches

SINGLE LAYER APPROACHES

2. Action recognition with space time trajectories• Campbell and Bobick

Curves in low-dimensional phase spaces

• Rao and Shah [2001]'s methodologyTheir system extracts meaningful curvature patterns from the trajectories.

Space-time Approaches

SINGLE LAYER APPROACHES

2. Action recognition with space time trajectories

Advantages• Ability to analyze detailed levels of human movements• View invariant methods

Space-time Approaches

SINGLE LAYER APPROACHES

3. Action recognition with space time features

• Chomat and Crowly• Calculates local probability of an activity• final recognition

• Rao and Shah [2001]'s methodology• An approach utilizing local spatio-temporal features at multiple temporal scales.

Multiple temporally scaled video volumes are analyzed to handle execution speed variations of an action.

Space-time Approaches

SINGLE LAYER APPROACHES

1. Space time volume:• Space-time approaches are suitable for recognition of periodic

actions and gestures, and many have been tested on public datasets.• Provides straight forward solution. • Often have difficulties in handling speed and motion variations

inherently.

2. Space-time trajectories

• Recognition approaches using space-time trajectories are able to perform detailed-level analysis and are view-invariant in most cases.

Comparison:Space-time Approaches

SINGLE LAYER APPROACHES

3. Spatio-temporal local feature-based approaches• Getting an increasing an amount of attention.• Recognize multiple activities without background subtraction or body-

part modeling.

LIMITATIONSThe major limitation of the space-time feature-based approaches is that

they are not suitable for modeling more complex activities. The relations among features are important for a non-periodic activity that takes a certain amount of time, which most of the previous approaches ignored.

Comparison:Space-time Approaches

SINGLE LAYER APPROACHES

1. Exemplar based2. State based

Sequential Approaches

SINGLE LAYER APPROACHES

1. Exemplar Based• Compare the input video with the template video.• DTW( Dynamic time warping ) algorithm is used for matching

variations. • Multiple cameras have been used to obtain 3-D body-part models of

a human, which is composed of a collection of segments and their joint angles.

Sequential Approaches

SINGLE LAYER APPROACHES

2. State model-based Approaches• Represent a human activity as a model composed

of a set of states. • An activity is represented in terms of a set of

hidden states. • A human is assumed to be in one state at each

time frame, and each state generates an observation.

Sequential Approaches

SINGLE LAYER APPROACHES

State model-based Approaches

The evaluation problem is a problem of calculating the probability of a given sequence (i.e. new input) generated by a particular state-model.

If the calculated probability is high enough, the state model-based approaches are able to decide that the activity corresponding to the model occurred in the givenInput.

Sequential Approaches

SINGLE LAYER APPROACHES

Comparison:• Enable to detect more complex activities like nom periodic activities.• Able to make a probabilistic analysis on the activity.• Calculates a posterior probability of an activity occurring, enabling it to be

easily incorporated with other decisions.

Sequential Approaches

HIERARICHAL APPROACHES

1. Statistical approaches

2. Syntactic approaches

3. Description-based approaches

HIERARICHAL APPROACHES

Statistical approaches•At the bottom layer, atomic actions are recognized from sequences of feature vectors, just as in single-layered sequential approaches. As a result, a sequence of feature vectors are converted to a sequence of atomic actions. For each model, a probability of the modelgenerating a sequence of observations (i.e. atomic-level actions) is calculated tomeasure the likelihood between the activity and the input image sequence.

HIERARICHAL APPROACHES

Statistical approaches

HIERARICHAL APPROACHES

Syntactic approaches

• Syntactic approaches model human activities as a string of symbols, where each symbol corresponds to an atomic-level action.• Require atomic-level actions to be recognized first, using any of the previous techniques

HIERARICHAL APPROACHES

Syntactic approaches

One of the limitations of syntactic approaches is in the recognition of concurrentactivities. Syntactic approaches are able to probabilistically recognize hierarchicalactivities composed of sequential sub-events, but are inherently limited on activitiescomposed of concurrent sub-events

HIERARICHAL APPROACHES

Description-based approaches

In description-based approaches, a time interval is usually associated with anoccurring sub-event to specify necessary temporal relationships among sub-events.

Seven basic predicates that Allen hasdened are: before, meets, overlaps, during, starts, nishes, and equals.

HIERARICHAL APPROACHES

Description-based approaches

HIERARICHAL APPROACHES

Comparison

•Suitable for recognizing high-level.•Easily incorporate human knowledge into the systems•Require less training data

1. Statistical and syntactic approacheso Provide a probabilistic framework for reliable recognition with noisy

inputs.2. Description-based approaches

o represent and recognize human activities with complex temporal structures.

o Sequentially and concurrent organized sub-events are handled.

HUMAN-OBJECT INTERACTIONS AND GROUP ACTIVITIES

EXTENDED PORTION OF THE PAPER

HUMAN-OBJECT INTERACTIONS

Integration of multiple components is required to recognize human object interactions

Steps involved:• Identification of objects • Motion involved in an activity•Analysis of their interplays

These components are highly dependent on each other.

The results suggest that the recognition of objects can benefit activity recognition while activity recognition helps the classification of objects.

HUMAN-OBJECT INTERACTIONS

Moore et al. [1999]

Compensates for the failures of object classification with the recognition results of simple actions.

Common Performance of system: object recognition estimates human activities with objects But can act conversely as well

Peursum et al. [2005]Focused on the fact that humans interact with objects in many different ways, depending on the function of the objectsObject recognition solely based on the activity information

HUMAN-OBJECT INTERACTIONS

proposed a probabilistic model integrating an objects' appearance, human motion with objects, and reactions of objects.

Two types of motion in which humans interact with objects, `reach motion' and `manipulation motion', are estimated.

Gupta and Davis [2007]

Ryoo and Aggarwal [2007]

Their object recognition and motion estimation components were constructedto help each other.compensate for object recognition failures or motion estimation failures.get feedback from the high-level activity recognition results for improved recognition.

GROUP ACTIVITIES

Group activities are the activities whose actors are one or more conceptual groups.

In order to recognize group activities, the analysis of activities of individuals as well as their overall relations becomes essential.

CONTAINS TWO FOCUSE POINTS1. Researchers have focused on the recognition of group activities where each group

member has its own role different from the others.2. The second type of group activity is the activities which are characterized by the

overall motion of entire group members.

CONCLUSION

•Applications of human activity recognition are diverse.

•Tracking and monitoring people is becoming and integral part of everyday activities.

•The paper gives the latest and the previous methodologies been explored.

•1999 , human activity recognition was in its infancy.

•Early cameras were fixed and simple.

•Today's cameras with pan-tilt-zoom features creates more challenges for the researchers.

• problem areas, causing failures: noise, lights, distance and tracking.

•Future direction is encouraged and dictated by applications.

REFERENCES

http://spie.org/x34279.xml?ArticleID=x34279

http://www.google.com.pk/imgres?q=application+of+cameras+in+public+places&um=1&hl=en&biw=1280&bih=656&tbm=isch&tbnid=1qN-Vew9MW10pM:&imgrefurl=http://www.securitynewsdaily.com/10-ways-government-watches-you-1103/5&docid=NCkbsDcxxjffnM&w=450&h=300&ei=QkeVTqbyGYi28QOlss2VBw&zoom=1

http://spie.org/Images/Graphics/Newsroom/Imported-2011/003455/003455_10_fig1.jpg

THANKS!