CMPT888–HumanActivityRecognition

GregMori

Outline

•  Introtoclass•  Administrativedetails

Overview

•  Thisclassisaboutvision‐basedactionrecognition–  Inputisimagesorvideos– Outputisdescriptionofwhatpeoplearedoingintheimages/videos

ActionRecognitionExample

•  Recognizehumanactionsfromrawvideodata

Gatheringactiondata

•  3components:– detecthumans,track,recognizeaction

6

ApplicationsI•  Automatedvideosurveillance– Drawattentiontoactionsofinterest– Savehumanoperatortime

ApplicationsII

•  Collectdataonpedestrianbehaviour–  CollaborationwithSaunierandSayed(UBCCivilEngineering)

ApplicationsIII

Automaticallydetectfalls,near‐falls(withS.RobinovitchSFU)

WhyuseComputerVision?

•  Competingapproaches– Wearablesensors

– Manuallabour

•  Non‐intrusive– Donotneedcooperativesubjects

•  Inexpensive,nooperatorfatigue– Semi‐automatictechniques

PROBLEMDEFINITION

WhatisActionRecognition?

•  Terminology– Whatisan“action”?

•  Outputrepresentation– Whatdowewanttosayaboutanimage/video?

Unfortunately,neitherquestionhassatisfactoryansweryet

Terminology

•  Theterms“actionrecognition”,“activityrecognition”,“eventrecognition”,areusedinconsistently– Findingacommonlanguagefordescribingvideosisanopenproblem

TerminologyExample

•  “Action”isalow‐levelprimitivewithsemanticmeaning–  E.g.walking,pointing,placinganobject

•  “Activity”isahigher‐levelcombinationwithsometemporalrelations–  E.g.takingmoneyoutfromATM,waitingforabus

•  “Event”isacombinationofactivities,ofteninvolvingmultipleindividuals–  E.g.asoccergame,atrafficaccident

•  Thisiscontentious– Nostandard,rigorousdefinitionexists

OutputRepresentation

•  Thisimagecontainsamanwalking– Actionclassification/recognition

•  Themanwalkingishere– Actiondetection

•  Giventhisimagewhatisthedesiredoutput?

OutputRepresentation

•  Thisimagecontains5menwalking,4jogging,2running

•  The5menwalkingarehere

•  Thisisasoccergame

•  Giventhisimagewhatisthedesiredoutput?

OutputRepresentation

•  Giventhisvideowhatisthedesiredoutput?

•  Frames1‐20themanrantotheleft,thenframes21‐25heranawayfromthecamera

•  Isthisanaccuratedescription?•  Arelabelsandvideoframesin1‐1correspondence?

ChallengesinRecognition

•  Intra‐classvariation•  Objectposevariation•  Backgroundclutter•  Occlusion•  Lighting

TRIMESTERPREVIEW

Week2

•  Preliminaries– Humandetection

– Backgroundsubtraction– Opticalflow

Dalal+TriggsCVPR05

Weeks3‐4

•  MotionTemplates

BobickandDavisPAMI01

Efrosetal.ICCV03

Weeks5‐6

•  Localfeaturevideorepresentations

Schuldtetal.ICPR04Dollaretal.VSPETS05

Week7

•  Unsupervisedandweaklysupervisedmethods

Laptevetal.CVPR08

Week8

•  Temporalmodels

?? ? ? ? ? ? ? ? ?

WangandMoriPAMI09

Week9

•  Humanposeestimationandposeretrieval

Yangetal.CVPR10

Week10

•  Discriminativemethods

Runright Walkleft Runright45

FathiandMoriCVPR08

Week11•  Humanactionsinstillimages

SLAGWangetal.CVPR06

ADMINISTRIVIA

CoursePlan

•  Readresearchpapers– ForeachtopicIpresentimportantpapers

– Studentseachpresentarecentpaper– Wediscuss

•  Doaproject– Gainin‐depthexperienceonaproblemandalgorithm

Introductions

Prerequisite

•  Noformalprerequisites– Butitwouldbebestifyouknowsomecomputervision/imageprocessingandsomemachinelearning

•  Youwillneedtodotheusualthings– Math(continuous),programming,reading,writing,presenting

•  Askmeifyouareconcerned

GradingScheme

•  10%Classparticipation–  Participateindiscussionsaboutpapers,ask/answerquestions

•  10%Readingassignments–  1or2paperseachweek;subsetoftheonesIpresent

•  10%Paperpresentation–  Choosefromlistofpapersonline

•  10%Assignment–  Smallprogrammingassignmentonmotionanalysis

•  60%Project–  Individualorinsmallgroups–  Presentation,writtenreport

ReadingAssignments

•  Similartominipaperreview– Oneparagraphsummarizingpaper

– Criticaldiscussion(whatyoulike/don’tlike)– Questionsyouhave(formetoexplain)

•  Duebeforestartoflectureviaemail– FirstonedueMonday

•  Thesedetailsandlistofpapersareonline

PaperPresentations

•  Chooseonepaperthatinterestsyou– Fromlistonline/insyllabus

•  20minutepresentation– 10+minutesquestions/discussion– Feelfreetouseslidesprovidedbyauthors

Assignment

•  Shortprogrammingassignment– Backgroundsubtraction– Motion‐basedactionrecognition

•  Outnextweek,due2weekslater

Project

•  Majorcomponentofcourse– Recognizeactions

•  Implementexistingtechnique– Orvariantthereof– Canusesomethingyou’reworkingoninyourresearch•  Mustrecognizeactions•  Mustdosomethingthatdidn’texistbeforethiscourse

•  Proposal,presentation,report

CoursePlan

•  Nextweek– Preliminaries

•  Backgroundsubtraction,humandetection,motion

•  Afterthat– Papers,papers,papers