Collective Activity Detection using Hinge-loss Markov Random Fields

Ben London, Sameh Khamis, Stephen H. Bach, Bert Huang, Lise Getoor, Larry DavisUniversity of Maryland

College Park, MD 20742{blondon,sameh,bach,bert,getoor,lsdavis}@cs.umd.edu

Abstract

We propose hinge-loss Markov random fields (HL-MRFs), a powerful class of continuous-valued graphicalmodels, for high-level computer vision tasks. HL-MRFsare characterized by log-concave density functions, and areable to perform efficient, exact inference. Their templatedhinge-loss potential functions naturally encode soft-valuedlogical rules. Using the declarative modeling languageprobabilistic soft logic, one can easily define HL-MRFs viafamiliar constructs from first-order logic. We apply HL-MRFs to the task of activity detection, using principles ofcollective classification. Our model is simple, intuitive andinterpretable. We evaluate our model on two datasets andshow that it achieves significant lift over the low-level de-tectors.

1. IntroductionIn many computer vision tasks, it is useful to combine

structured, high-level reasoning with low-level predictions.Collective reasoning at a high-level can take advantage ofaccurate low-level detectors, while improving the accuracyof predictions based on less accurate detectors. To fullyleverage the power of high-level reasoning, we require atool that is both powerful enough to model complex, struc-tured problems and expressive enough to easily encodehigh-level ideas. In this work, we apply hinge-loss Markovrandom fields (HL-MRFs) [2, 3] to a high-level vision task.HL-MRFs are powerful, templated graphical models thatadmit efficient, exact inference over continuous variables.We demonstrate that, when combined with the modelinglanguage probabilistic soft logic (PSL) [7, 17], HL-MRFsallow us to design high-level, structured models that im-prove the performance of low-level detectors.

We focus on the task of collective activity detection ofhumans in video scenes. Since human activities are ofteninteractive or social in nature, collective reasoning over ac-tivities can provide more accurate detections than indepen-dent, local predictions. For instance, one can use aggregate

predictions within the scene or frame to reason about thelocal actions of each actor. Further, collective models letus reason across video frames, to allow predictions in ad-jacent frames to inform each other, and thus implement theintuition that actions are temporally continuous.

We demonstrate the effectiveness of HL-MRFs and PSLon two group activity datasets. Using a simple, interpretablemodel, we are able to achieve significant lift in accuracyfrom low-level predictors. We thus show HL-MRFs to be apowerful, expressive tool for high-level computer vision.

1.1. Related Work

Motivated by the rich spatiotemporal structure of humanactivity, recent work in high-level computer vision has fo-cused on modeling the complex interactions among obser-vations explicitly, solving multiple vision problems jointly.These interactions could be between scenes and actions[21], objects and actions [13, 28], or actions performed bytwo or more people [8, 9, 18, 19]. They have been modeledusing context-free grammars [25], AND-OR graphs [1, 14],probabilistic first-order logic [6, 22], and as network flow[10, 15, 16].

While most of these approaches require that personbounding boxes are pre-detected and pre-tracked to incor-porate temporal cues [6, 8, 9, 13, 14, 18, 22, 25], recentwork proposes solving activity recognition and trackingjointly. Khamis et al. [15] presented a network flow modelto perform simultaneous action recognition and identitymaintenance. They then augmented their model to jointlyreason about scene types [16]. Similarly, Choi and Savarese[10] proposed a unified model to perform action recogni-tion at the individual and group levels simultaneously withtracking. We build upon this work using a probabilistic re-lational approach.

PSL is one of many existing systems for probabilis-tic relational modeling, including Markov logic networks[24], relational dependency networks [23], and relationalMarkov networks [27], among others. One distinguishingfeature of PSL is that its continuous representation of log-ical truth makes its underlying probabilistic model an HL-

2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops

978-0-7695-4990-3/13 $26.00 © 2013 IEEE

DOI 10.1109/CVPRW.2013.157

566

2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops

978-0-7695-4990-3/13 $26.00 © 2013 IEEE

DOI 10.1109/CVPRW.2013.157

566

2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops

978-0-7695-4990-3/13 $26.00 © 2013 IEEE

DOI 10.1109/CVPRW.2013.157

566

2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops

978-0-7695-4990-3/13 $26.00 © 2013 IEEE

DOI 10.1109/CVPRW.2013.157

566

MRF [3], which allows inference of the most-probable ex-planation (MPE) to be solved as a convex optimization. Ourwork benefits from recent advances on fast HL-MRF infer-ence based on the alternating direction method of multipli-ers [2, 5], which significantly increases the scalability ofHL-MRF inference over off-the-shelf convex optimizationtools.

2. Hinge-loss Markov Random FieldsIn this section we present hinge-loss Markov ran-

dom fields (HL-MRFs), a general class of conditional,continuous-valued probabilistic models. HL-MRFs are log-linear probabilistic models whose features are hinge-lossfunctions of the variable states. Through constructionsbased on soft logic (explained in Section 3), hinge-loss po-tentials can be used to model generalizations of logical con-junction and implication, making these powerful models in-terpretable, flexible, and expressive.

HL-MRFs are parameterized by constrained hinge-lossenergy functions.

Definition 1. Let Y = (Y1, . . . , Yn) be a vector of n vari-ables and X = (X1, . . . , Xn′) a vector of n′ variables withjoint domain D = [0, 1]n+n′

. Let φ = (φ1, . . . , φm) be mcontinuous potentials of the form

φj(Y,X) = [max {`j(Y,X), 0}]pj

where `j is a linear function of Y and X and pj ∈ {1, 2}.Let C = (C1, . . . , Cr) be linear constraint functions asso-ciated with index sets denoting equality constraints E andinequality constraints I, which define the feasible set

D̃ =

{Y,X ∈ D

∣∣∣∣ Ck(Y,X) = 0,∀k ∈ ECk(Y,X) ≥ 0,∀k ∈ I

}.

For Y,X ∈ D̃, given a vector of nonnegative free param-eters, i.e., weights, λ = (λ1, . . . , λm), a constrained hinge-loss energy function fλ is defined as

fλ(Y,X) =

m∑j=1

λjφj(Y,X) .

Definition 2. A hinge-loss Markov random field P overrandom variables Y and conditioned on random variablesX is a probability density defined as follows: if Y,X /∈ D̃,then P (Y|X) = 0; if Y,X ∈ D̃, then

P (Y|X) =1

Z(λ)exp [−fλ(Y,X)] , (1)

where Z(λ) =∫Y

exp [−fλ(Y,X)].

The potential functions and weights can be grouped to-gether into templates, which are used to define general

classes of HL-MRFs that are parameterized by the struc-ture of input data. Let T = (t1, . . . , ts) denote a vector oftemplates with associated weights Λ = (Λ1, . . . ,Λs). Wepartition the potentials by their associated templates and let

Φq(Y,X) =∑j∈tq

φj(Y,X)

for all tq ∈ T . In the ground HL-MRF, the weight of thej’th hinge-loss potential is set to the weight of the templatefrom which it was derived, i.e., λj = Λq , for each j ∈ tq .

MPE inference in HL-MRFs is equivalent to finding thefeasible minimizer of the convex energy fλ. Here, HL-MRFs have a distinct advantage over general discrete mod-els, since minimizing fλ is a convex optimization, ratherthan a combinatorial one.

Bach et al. [2] showed how to minimize fλ using aconsensus-optimization algorithm, based on the alternatingdirection method of multipliers (ADMM) [5]. Consensus-optimization works by creating local copies of the variablesin each potential and constraint, constraining them to beequal to the original variables, and relaxing those equal-ity constraints to make independent subproblems. By it-eratively solving the subproblems and averaging the results,the algorithm reaches a consensus on the best values of theoriginal variables, also called the consensus variables. Thisprocedure is guaranteed to converge to the global minimizerof fλ. (See [5] for more details on consensus optimizationand ADMM.)

The inference algorithm has since been generalized andimproved [3]. Experimental results suggest that the runningtime of this algorithm scales linearly with the size of theproblem. On modern hardware, the algorithm can performexact MPE inference with hundreds of thousands of vari-ables in just a few seconds.

2.1. Weight Learning

To learn the parameters Λ of an HL-MRF given a set oftraining examples, we perform maximum-likelihood estima-tion (MLE), using the voted perceptron algorithm [11]. Thepartial derivative of the log of Equation 1 with respect to aparameter Λq is

∂ logP (Y|X)

∂Λq= EΛ [Φq(Y,X)]− Φq(Y,X) , (2)

where EΛ is the expectation under the distribution definedby Λ. Note that the expectation in Equation 2 is intractableto compute. To circumvent this, we use a common approx-imation: the values of the potential functions at the mostprobable setting of Y with the current parameters [7]. TheMPE approximation of the expectation is fast, due to thespeed of the inference algorithm; however, there are noguarantees about its quality.

567567567567

The voted perceptron algorithm optimizes Λ by takingsteps of fixed length in the direction of the negative gradi-ent, then averaging the points after all steps. To preservethe nonnegativity of the weights, any step that is outside thefeasible region is projected back before continuing. For asmoother ascent, it is often helpful to divide the q-th com-ponent of the gradient by the number of groundings |tq| ofthe q’th template [20], which we do in our experiments.

3. Probabilistic Soft LogicIn this section, we review probabilistic soft logic (PSL)

[7, 17], a declarative language for probabilistic reasoning.While PSL borrows the syntax of first-order logic, semanti-cally, all variables take soft truth values in the interval [0, 1],instead of only the extremes, 0 (FALSE) and 1 (TRUE). Con-tinuous variables are useful both for modeling continuousdomains as well as for expressing confidences in discretepredictions, which are desirable for the same reason thatpractitioners often prefer marginal probabilities to discreteMPE predictions. PSL provides a natural interface to designhinge-loss potential templates using familiar concepts fromfirst-order logic.

A PSL program consists of a set of first-order logicrules with conjunctive bodies and disjunctive heads. Rulesare constructed using the logical operators for conjunction(∧), negation (¬) and implication (⇒). Rules are assignedweights, which can be learned from observed data. Con-sider the following rule for collective image segmentation.

0.8 : CLOSE(P1, P2) ∧ LABEL(P1, C)⇒ LABEL(P2, C)

In this example, P1, P2 and C are variables representingtwo pixels and a category; the predicate CLOSE(P1, P2)measures the degree to which P1, P2 are “close” in the im-age; LABEL(P1, C) indicates the degree to which P1 be-longs to class C, and similarly for LABEL(P2, C). Thisrule has weight 0.8.

PSL uses the Lukasiewicz t-norm, and its correspond-ing co-norm, to relax the logical operators for continuousvariables. These relaxations are exact at the extremes, butprovide a consistent mapping for values in between. Forexample, given variables X and Y , the relaxation of theconjunction X ∧ Y would be max{0, X + Y − 1}.

We say that a rule r is satisfied when the truth value ofthe head rHEAD is at least as great as that of the body rBODY.The rule’s distance from satisfaction dr measures the degreeto which this condition is violated:

dr = max{0, rBODY − rHEAD}. (3)

This corresponds to one minus the truth value of rBODY ⇒rHEAD when the variables are {0, 1}-valued. In the processknown as grounding, each rule is instantiated for all possi-ble substitutions of the variables as given by the data. For

example, the above rule would be grounded for all pairs ofpixels and categories.1

Notice that Equation 3 corresponds to a convex hingefunction. In fact, each rule corresponds to a particular tem-plate t ∈ T , and each grounded rule corresponds to a poten-tial in the ground HL-MRF. If we let Xi,j denote the close-ness of pixels pi, pj , and Yi,c denote the degree to which pihas label c (likewise for pj), then the example rule abovewould correspond to the potential function

φ(Y,X) = [max{0, Xi,j + Yi,c − Yj,c − 1}]p,

where p ∈ {1, 2} is the exponent parameter (see Defini-tion 1). Thus, PSL, via HL-MRFs, defines a log-lineardistribution over possible interpretations of the first-orderrules.

Because it is backed by HL-MRFs, PSL has some addi-tional features that are useful for modeling. The constraintsin Definition 1 allow the encoding of functional modelingrequirements, which can be used to enforce mutually exclu-sion constraints (i.e., that the soft-truth values should sumto one). Further, the exponent parameter p allows flexibil-ity in the shape of the hinge, affecting the sharpness of thepenalty for violating the logical implication. Setting p to 1penalizes violation linearly with the amount the implicationis unsatisfied, while setting p to 2 penalizes small violationsmuch less. In effect, some linear potentials overrule oth-ers, while the influences of squared potentials are averagedtogether.

4. Collective Activity DetectionIn this section, we apply HL-MRFs to the task of collec-

tive activity detection. We treat this as a high-level visiontask, using the output of primitive, local models as input toa collective model for joint reasoning. We begin by describ-ing the datasets and objective. We then describe our model.We conclude with a discussion of our experimental results.

4.1. Datasets

We use the collective activity dataset from [8] and itsaugmentation from [9] to evaluate our model. The firstdataset contains 44 video sequences, each containing mul-tiple actors performing activities in the set: crossing, stand-ing, queueing, walking, and talking. The second datasetcontains 63 sequences, with actions in: crossing, stand-ing, queueing, talking, dancing, and jogging.2 From eachdataset, we use the bounding boxes (with position, widthand height), pixel data, actions and identity annotations; we

1Though this could possibly lead to an explosion of groundings, PSLuses lazy activation to only create groundings for substitutions when thetruth value of the body exceeds a certain margin.

2The walking action was removed from the augmented dataset by [9]because it was deemed ill-defined.

568568568568

crossing waiting queueing walking talking dancing jogging

Figure 1. A few sample frames from the collective activity datasets. The original dataset and its augmentation include multiple actorsin a natural setting performing specific actions. The colors of the bounding boxes in the figure specify the groundtruth action of thecorresponding person.

do not use the 3-D trajectories. Activity detection in thesedatasets is challenging, since the scenes involve multipleactors in a natural setting; other action datasets, like KTH[26] or Weizmann [4], have a single person performing aspecific action. In addition, there is considerable ambigu-ity in the actions being considered; for example, the actionsstanding and queueing are difficult to distinguish, even fora human. Figure 1 illustrates some sample frames from thetwo datasets.

Similar to [15, 16], we represent the detected human fig-ures using histogram of oriented gradients (HOG) [12] fea-tures and action context (AC) descriptors [18]. The AC de-scriptor is a feature representation that combines the localbeliefs about an actor’s activities with those of actors in sur-rounding spatiotemporal neighborhoods. To create the ACdescriptors, we use HOG features as the underlying featurerepresentation; we then train a first-level SVM classifier onthese features and combine the outputs per [18]. Finally, wetrain a second-stage SVM classifier on the AC descriptors toobtain the activity beliefs used in our high-level model. Allclassifiers are trained using a leave-one-out methodology,such that the predictions for the i’th sequence are obtainedby training on all other sequences.

4.2. Model

Our primary objective is to enhance the low-level activ-ity detectors with high-level, global reasoning. To do so,we augment the local features (described below) using re-

lational information within and across adjacent frames. Bymodeling the relationships of bounding boxes, we can lever-age certain intuitions about human activity. For instance, itis natural to assume that one’s activity is temporally contin-uous; that is, it is not likely to change between points closein time. Further, there are certain activities that involve in-teraction with others, such as talking or queueing. There-fore, if we believe that one or more actors are talking, thenactors nearby are also likely to be talking. Using PSL, mod-eling these intuitions is a simple matter of expressing themin first-order logic. We can then use HL-MRFs to reasonjointly over these rules.

Our PSL model is given below.

LOCAL(B, a) ⇒ DOING(B, a) (R1)

FRAME(B,F ) ∧ FRAMELABEL(F, a) ⇒ DOING(B, a) (R2)

CLOSE(B1, B2) ∧ DOING(B1, a) ⇒ DOING(B2, a) (R3)

SEQ(B1, B2) ∧ CLOSE(B1, B2) ⇒ SAME(B1, B2) (R4)

SAME(B1, B2) ∧ DOING(B1, a) ⇒ DOING(B2, a) (R5)

Rule R1 corresponds to beliefs about local predictions (oneither the HOG features or AC descriptors). R2 expressesthe belief that if many actors in the current frame are do-ing a particular action, then perhaps everyone is doing thataction. To implement this, we derive a FRAMELABEL pred-icate for each frame; this is computed by accumulating andnormalizing the LOCAL activity beliefs for all actors in theframe. Similarly, R3 enforces our intuition about the ef-

569569569569

fect of proximity on activity, where actors that are close3

in the same frame are likely to perform the same action.This can be considered a fine-grained version of the sec-ond rule. R4 is used for identity maintenance and tracking.It essentially says that if two bounding boxes occur in ad-jacent frames and their positions have not changed signifi-cantly, then they are likely the same actor. We then reason,in R5, that if two bounding boxes (in adjacent frames) referto the same actor, then they are likely to be doing the sameactivity. Note that rules involving lowercase a are definedfor each action a, such that we can learn different weightsfor different actions. We define priors over the predicatesSAME and DOING, which we omit for space. We also de-fine (partial) functional constraints (not shown), such thatthe truth-values over all actions (respectively, over all adja-cent bounding boxes), sum to (at most) one. We train theweights for these rules using 50 iterations of voted percep-tron, with a step size of 0.1.

Note that we perform identity maintenance only to im-prove our activity predictions. During prediction, we do notobserve the SAME predicate, so we have to predict it. Wethen use these predictions to inform the rules pertaining toactivities.

4.3. Experiments

To illustrate the lift one can achieve on low-level predic-tors, we evaluate two versions of our model: the first usesactivity beliefs from predictions on the HOG features; thesecond uses activity beliefs predicted on the AC descrip-tors. Essentially, this determines which low-level predic-tions are used in the predicates LOCAL and FRAMELA-BEL. We denote these models by HL-MRF + HOG and HL-MRF + ACD respectively. We compare these to the pre-dictions made by the first-stage predictor (HOG) and thesecond-stage predictor (ACD).

The results of these experiments are listed in Table 1. Wealso provide recall matrices (row-normalized confusion ma-trices) for HL-MRF + ACD in Figure 2. For each dataset,we use leave-one-out cross-validation, where we train ourmodel on all except one sequence, then evaluate our pre-dictions on the hold-out sequence. We report cumulativeaccuracy and F1 to compensate for skew in the size and la-bel distribution across sequences; this involves accumulat-ing the confusion matrices across folds.

Our results illustrate that our models are able to achievesignificant lift in accuracy and F1 over the low-level detec-tors. Specifically, we see that HL-MRF + HOG achieves a12 to 20 point lift over the baseline HOG model, and HL-MRF + ACD obtains a 1.5 to 2.5 point lift over the ACDmodel.

3To measure closeness, we use an RBF kernel.

Table 1. Results of experiments with the 5- and 6-activity datasets,using leave-one-out cross-validation. The first dataset contains 44sequences; the second, 63 sequences. Scores are reported as thecumulative accuracy/F1, to account for size and label skew acrossfolds.

5 Activities 6 ActivitiesMethod Acc. F1 Acc. F1HOG .474 .481 .596 .582HL-MRF + HOG .598 .603 .793 .789ACD .675 .678 .835 .835HL-MRF + ACD .692 .693 .860 .860

Figure 2. Recall matrices (i.e., row-normalized confusion matri-ces) for the 5- and 6-activity datasets, using the HL-MRF + ACDmodel.

5. Conclusion

We have shown that HL-MRFs are a powerful class ofmodels for high-level computer vision tasks. When com-bined with PSL, designing probabilistic models is easy andintuitive. We applied these models to the task of collec-tive activity detection, building on local, low-level detectorsto create a global, relational model. Using simple, inter-pretable first-order logic rules, we were able to improve theaccuracy of low-level detectors.

570570570570

Acknowledgements

This work was partially supported by NSF grantsIIS1218488 and CCF0937094, NSF CAREER grant0746930, and MURI from the Office of Naval Research un-der grant N00014-10-1-0934.

References[1] M. R. Amer, D. Xie, M. Zhao, S. Todorovic, and S. C. Zhu,

“Cost-sensitive top-down/bottom-up inference for multiscaleactivity recognition,” in European Conference on ComputerVision, 2012. 1

[2] S. Bach, M. Broecheler, L. Getoor, and D. O’Leary, “ScalingMPE inference for constrained continuous markov randomfields with consensus optimization,” in Neural InformationProcessing Systems, 2012. 1, 2

[3] S. H. Bach, B. Huang, B. London, and L. Getoor, “Hinge-loss markov random fields: Convex inference for structuredprediction,” in Uncertainty in Artificial Intelligence, 2013. 1,2

[4] M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri,“Actions as space-time shapes,” in ICCV, 2005. 4

[5] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Dis-tributed optimization and statistical learning via the alternat-ing direction method of multipliers,” Foundations and Trendsin Machine Learning, vol. 3, no. 1, 2011. 2

[6] W. Brendel, S. Todorovic, and A. Fern, “Probabilistic eventlogic for interval-based event recognition,” in CVPR, 2011.1

[7] M. Broecheler, L. Mihalkova, and L. Getoor, “Probabilis-tic similarity logic,” in Uncertainty in Artificial Intelligence,2010. 1, 2, 3

[8] W. Choi, K. Shahid, and S. Savarese, “What are they doing?:Collective activity classification using spatio-temporal rela-tionship among people,” in VS, 2009. 1, 3

[9] W. Choi, K. Shahid, and S. Savarese, “Learning context forcollective activity recognition,” in CVPR, 2011. 1, 3

[10] W. Choi and S. Savarese, “A unified framework for multi-target tracking and collective activity recognition,” in Euro-pean Conference on Computer Vision, 2012. 1

[11] M. Collins, “Discriminative training methods for hiddenMarkov models: theory and experiments with perceptron al-gorithms,” in Empirical Methods in Natural Language Pro-cessing, 2002. 2

[12] N. Dalal and B. Triggs, “Histograms of oriented gradientsfor human detection,” in CVPR, 2005. 4

[13] A. Gupta and L. S. Davis, “Objects in action: An approachfor combining action understanding and object perception,”in CVPR, 2007. 1

[14] A. Gupta, P. Srinivasan, J. Shi, and L. S. Davis, “Understand-ing videos, constructing plots learning a visually groundedstoryline model from annotated videos,” in CVPR, 2009. 1

[15] S. Khamis, V. I. Morariu, and L. S. Davis, “A flow model forjoint action recognition and identity maintenance,” in CVPR,2012. 1, 4

[16] S. Khamis, V. I. Morariu, and L. S. Davis, “Combining per-frame and per-track cues for multi-person action recogni-tion,” in European Conference on Computer Vision, 2012.1, 4

[17] A. Kimmig, S. Bach, M. Broecheler, B. Huang, andL. Getoor, “A short introduction to probabilistic soft logic,”in NIPS Workshop on Probabilistic Programming: Founda-tions and Applications, 2012. 1, 3

[18] T. Lan, Y. Wang, G. Mori, and S. N. Robinovitch, “Retriev-ing actions in group contexts,” in SGA, 2010. 1, 4

[19] T. Lan, Y. Wang, W. Yang, and G. Mori, “Beyond actions:Discriminative models for contextual group activities,” inNIPS, 2010. 1

[20] D. Lowd and P. Domingos, “Efficient weight learning forMarkov logic networks,” in Principles and Practice ofKnowledge Discovery in Databases, 2007. 3

[21] M. Marszalek, I. Laptev, and C. Schmid, “Actions in con-text,” in CVPR, 2009. 1

[22] V. I. Morariu and L. S. Davis, “Multi-agent event recognitionin structured scenarios,” in CVPR, 2011. 1

[23] J. Neville and D. Jensen, “Relational dependency networks,”Journal of Machine Learning Research, vol. 8, pp. 653–692,2007. 1

[24] M. Richardson and P. Domingos, “Markov logic networks,”Machine Learning, vol. 62, no. 1-2, pp. 107–136, 2006. 1

[25] M. S. Ryoo and J. K. Aggarwal, “Stochastic representationand recognition of high-level group activities,” IJCV, vol. 93,no. 2, pp. 183–200, 2010. 1

[26] C. Schuldt, I. Laptev, and B. Caputo, “Recognizing humanactions: A local svm approach,” in ICPR, 2004. 4

[27] B. Taskar, M. Wong, P. Abbeel, and D. Koller, “Link pre-diction in relational data,” in Neural Information ProcessingSystems, 2003. 1

[28] B. Yao and L. Fei-Fei, “Modeling mutual context of ob-ject and human pose in human-object interaction activities,”CVPR, 2010. 1

571571571571