A Facial Affect Mapping Engine (FAME)

Leonardo Impett, Tadas Baltrusaitis, Peter RobinsonComputer Laboratory

University of Cambridge, Cambridge, UKli222@cam.ac.uk, tb346@cam.ac.uk, pr10@cam.ac.uk

(a) Puppeteer (b) Avatar (c) Final image

Figure 1: (a) The puppeteer controlling the animation showing the automatic facial landmark and head pose tracking. (b) Theavatar image being animated. (c) The final result of avatar being overlaid on the puppeteers image.

ABSTRACTFacial expressions play a crucial role in human interaction.Interactive digital games can help teaching people to both ex-press and recognise them. Such interactive games can benefitfrom the ability to alter user expressions dynamically and inreal-time. This is especially important for people that have animpaired ability to understand and display facial expression,such as people with Autism Spectrum Disorders.

In this paper, we present the Facial Affect Mapping Engine(FAME), a framework for mapping and manipulating facialexpression in real-time across images and video streams. Oursystem is fully automatic, and takes advantage of severalcomputationally inexpensive methods to work in real-time,on generic hardware. FAME presents new possibilities for thedesigners of intelligent interactive digital games. We compareour approach with previous similar systems demonstrating itsadvantages. We also suggest some use-case scenarios enabledby our framework scenarios.

Author KeywordsAugmented Reality; Facial Affect; Face Tracking; FacialPuppetry; Face Swapping; Intelligent Games.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.IDGEI’14, Haifa, Israel, February 24, 2014

ACM Classification KeywordsH.5.1. Multimedia Information Systems

1. INTRODUCTIONReal-time augmented and virtual reality (AR and VR) sys-tems are important tools in the creation of intelligent digi-tal games. Often these are used to model static solid objectsinteractively, for instance to facilitate play in children withautism spectrum conditions (ASC) [11]. However, the abilityto manipulate dynamic and non-rigid objects such as the faceand head is more difficult, and an ability to do so would beparticularly useful for AR and VR.

Figure 2: The role of animator and avatar in the system. Theuser can be either or both of these

The transposition of facial emotion and identity is particu-larly relevant to video-interactive intelligent games. Facialexpression is important in the communication of emotions,conversational turn-taking and empathy [8, 13]. In this pa-per, we present the Facial Affect Mapping Engine (FAME), aframework for animating unlabelled static images with videostreams, or more generally, modifying mapping facial expres-sion in real time across images and video streams (see Figure

1). Our framework allows for facial expression and –headpose to be modified dynamically as it is mapped from oneface (the animator) to another (the avatar) (see Figure 2).

This leads to three useful modes of operation; well-describedas an interaction between a user, captured by web-cam, andan unlabelled reference source image or video, which couldbe from a dataset or synthesised. These two video streamsswap roles as animators or avatars:

(a) A user live-animates a face from a source image. Thereference face (avatar) is then blended onto the user’s video-stream. The user’s expression and head pose are transferredto the character in the source image.

(b) The source video animates the user’s face. For non-extreme head movements, that animated face is again overlaidon the user’s video-stream, so that the source video’s expres-sion and head pose are transferred to the user.

(c) The user’s facial expression and head pose is exaggerated,attenuated or altered dynamically, then transferred back to theuser’s video stream. Only one video-stream is used; the iden-tity and expression are both from the same person.

2. RELATED WORKA fundamental part of this system is avatar animation, knownas facial puppetry, where a digital avatar is animated by apuppeteer through live video. Traditional facial puppetry sys-tems are two-stage, based on face tracking and subsequentexpression transfer. Saragih et al. [15] created a facial pup-petry system that runs in real-time and requires no labour-expensive person-specific models, requiring only a single un-labelled avatar image in neutral expression.

A similar facial puppetry system has been used for a psy-chological experiment that explored the effect on attenuationof head-pose and facial expression in photo-realistic video-conferencing scenarios [4]. In this system, the cropped faceswere shown to participants with no background. Our sys-tem enables researches to conduct similar experiments, withnatural and realistic blending of the modified face onto theoriginal video background.

Face swapping is also of relevance to this system, and re-search on this has tended to focus on image-based manipu-lation [3]. Successful systems for replacing faces in single-camera video have been developed [7, 17], though mainlybased around complex and computationally expensive 3Dmodels, rendering near-photo-realistic videos offline, or on-line using 3D video from a stereo scanner [18]. These offlineor specialist-hardware implementations are unsuitable for usein interactive intelligent games; our system runs in real-timeon conventional home computing hardware.

Video face swapping has been demonstrated by McDonaldand Castro [5] in real time, focussing on identity swappingrather than expression transfer. Their technique involveswarping a face-image directly over a live feed, and interpo-lating the average pixel values between the warped face andthe original feed; their system shows some significant light-ing and texture imperfections. A sample is shown from their

system in Figure 4.

Our approach aims to place more focus on realistic re-synthesis of the video through computationally inexpensivetechniques. In order to be used in intelligent games, the sys-tem must satisfy the following requirements:

1. Fully automatic (no training and set-up at consumer end)

2. Real-time

3. Run on commodity hardware

4. Realistic for user interaction

Each of the above systems meets some, but not all, of theserequirements. Our approach aims to place more focus on real-istic re-synthesis of the video through computationally inex-pensive techniques. We incorporate the ideas from previouswork to specifically meet these requirements for intelligentgames.

Figure 4: A comparison of McDonald and Castro’s system(left) with ours (right)

3. ARCHITECTUREThe Facial Affect Mapping Engine uses techniques from bothfacial puppetry and face swapping to transfer identity and ex-pression through video streams. Our system, unlike existingsystems, uses three stages (see Figure 3):

1. Face tracking

2. Expression manipulation

3. Video re-synthesis

The user live-video and reference video can switch roles as’puppet/avatar’ or ’puppeteer/animator’. Face tracking op-erates on both puppet and puppeteer in parallel, whilst ex-pression manipulation combines the two, along with any ex-pression modification, to make an animated puppet. Finally,the output video is synthesised by re-situating the animatedpuppet in either the user or reference video-streams. This iseffectively a face-swapping operation.

3.1 Face TrackingIn order to track the geometry of the face, we need to tracklandmark points on the face together with the head pose.For tracking faces we use a Constrained Local Neural Field(CLNF) model [1], an instance of the Constrained LocalModel (CLM) framework [14, 6]. CLNF model we use tracks66 feature points in the face, including: face outline, outlineof the eyes, eyebrows, nose outline, and inner and outer out-lines of the lips.

Figure 3: The system architecture; from unlabelled video input to affect-mapped output

CLNF TrackerThe CLNF model is parametrised using p = [s,R,q, t]terms, that can be varied to acquire various instances of themodel: the scale factor s; object rotation R (first two rowsof a 3D rotation matrix); 2D translation t; and a vector de-scribing non-rigid variation of the identity shape q. Our pointdistribution model (PDM) is:

xi = s ·R(xi + Φiq) + t. (1)

Here xi = (x, y) denotes the 2D location of the ith featurepoint in an image, xi = (X,Y, Z) is the mean value of theith element of the PDM in the 3D reference frame, and thevector Φi is the ith eigenvector obtained from the training setthat describes the linear variations of non-rigid shape of thisfeature point. The PDM was constructed from the Multi-PIEdataset using non-rigid structure from motion [16].

In CLNF we estimate the maximum a posteriori probability(MAP) of the face model parameters p:

p(p|{li=1}ni=1, I) ∝ p(p)

n∏i=1

p(li=1|xi, I), (2)

where li ∈ {1,−1} is a discrete random variable indicatingif the ith feature point is aligned or misaligned, p(p) is theprior probability of the model parameters p, and

∏ni=1 p(li =

1|xi, I) is the joint probability of the feature points x beingaligned at a particular point xi, given an intensity image I.

Patch experts are used to calculate p(li = 1|xi, I), which isthe probability of a feature being aligned at point xi (Equation1). The tracker used in this work can either use an SVR basedpatch expert, or a more accurate Local Neural Field [1], thatmodels non-linear relationships. This allows for a trade-off:faster but less accurate or slower but more accurate tracking.

We employ a common two step CLM fitting strategy [6, 14];performing an exhaustive local search around the current esti-mate of feature points leading to a response map around everyfeature point, and then iteratively updating the model param-eters to maximise Equation 2 until a convergence metric is

reached. For fitting we use Non-Uniform Regularised Land-mark Mean-Shift [1]. We employ a multi-scale approach forincreased robustness and accuracy.

As a prior p(p) for parameters p, we assume that the non-rigid shape parameters q vary according to a Gaussian distri-bution with the variance of the ith parameter corresponding tothe eigenvalue of the ith mode of non-rigid deformation; therigid parameters s,R, and t follow a non-informative uni-form distribution.

3.2 Expression ManipulationIn this stage, we extract the faces from both images, modifythe expression in the puppeteer’s face and map that expres-sion onto the puppet’s face. Having tracked the important fea-ture points in the puppet’s and puppeteer’s video streams, weextract the tracked face image from the whole image frameby cropping. We then triangulate the face images with thepoints tracked, and piecewise affine-warp these faces to a neu-tral shape. This neutral-shape face image is then convertedfrom the Red, Green, Blue (RGB) colour space to a floating-point representation in the Luma-Chrominance (YUV) colourspace, represented by an achromatic brightness and two chro-matic components. The luminance is normalised to facilitatesubsequent comparison between expressions, and the YUVhistograms are equalised to give better robustness against thevariation of ambient lighting conditions.

At this point, the expression in a video source has been sep-arated into a texture (image) and shape (triangular mesh).Changes in expression influence both texture and shape, butthese can be controlled separately at this point.

The expression in the face shape can be modified throughmanipulation of the expression vector (p). We can attenu-ate or exaggerate facial expression by scaling the elementsof the non-rigid shape vector (q). We can also attenuate orexaggerate rotation and translation of the head by manipu-lating the rigid shape parameters t, R, s. More sophisticatedexpression modification can be used to control specific fa-

cial regions or expressions associated with specific emotions.Figure 5 shows an example of the expression modificationpossible with this technique.

Figure 5: Attentuation of expression using the expressionvector. Left to right: expression amplitude = 0.0, 0.5, 1.0

Facial expression and head pose both produce changes in theface texture, due to changes in illumination, wrinkles and soon. For expression to be mapped realistically, these textu-ral changes must also be present in the animated face. Thedynamic features (illumination, wrinkles) from the puppeteertexture must be combined with the static features (identity)from the puppet texture.

Previous systems have used dense 3D mesh re-lighting tech-niques [18] or synthesised textures based on active appear-ance models [15]. Both methods are too computationally ex-pensive for our purposes. We therefore apply a video expres-sion ratio image (ERI), based on the approach outlined by Liuet al. [12]. The face images are manipulated in four stages,as illustrated in Figure 6:

1. We capture a single neutral-expression face image of thepuppeteer. We then map that face image to a standard shape,so that we can compare changes in texture between this andsubsequent face images.

2. We blur both the neutral face and the current puppeteer faceto remove very fine noise, especially to do with misalignmentof features despite warping to a common standard shape. Forcomputational efficiency, we downsample the image at thispoint.

3. We create a new matrix, the luminance-component Expres-sion Ratio Image (ERI), of the same size of the faces: eachpixel within is the floating-point division of the correspond-ing pixels in the current puppeteer face by the pixel in theneutral face.

4. This ERI matrix is then used as a pixel-by-pixel multipli-cation factor for the neutrally-warped target avatar face (ef-fectively making wrinkle and shadow areas darker).

If we take A to be the neutral facial image of person A (thepuppeteer), A’ to be their face image with current expression,and B to be the neutral avatar face, then the animated avatarB’ can be expressed as:

B′(u, v) =B(u, v) ·A′(u, v)

A(u, v)(3)

If face tracking is successful, ERI leads to very believableresults; however, slight misalignments in facial tracking caneasily cause spurious results. For instance, if a mole or darkspectacles deviate in position slightly from in the ’neutral’

texture, the ERI will lead to high-luminance ’ghosts’ aroundtheir original positions. Similarly, changes in luminancearound the lips and hairline can be very sharp, and can lead tosimilar artefacts. Although slight misalignment is made tol-erable by the blurring phases, the trade-off between loss ofdetail and sensitivity of re-alignment is unfavourable.

In order to counter these effects, we limit the domain ofthe ERI to only pixels which get darker (shadows, wrinklesand frown-lines). In fact, we found that almost no visualexpression information was lost by discarding the ’lighten-ing’ data. The amplitude of this ERI is then attenuated by adynamically-controllable weight factor (initially 0.33, whichgave good results by visual inspection).

Figure 6: Our four-stage ERI system (left to right): the orig-inal puppeteer image; the puppeteer image warped to a stan-dard shape; the resulting ERI map; ERI texture transferredonto the puppet image

3.3 Face re-synthesisTo resynthesise the face, the expression avatar texture iswarped to the modified shape of the puppeteer. Hardware-assisted graphics is used to blend the new face and the back-ground image. Previous offline photo and video systems [17]have used Poisson blending techniques to fit the face into theimage without inconsistencies in lighting or colour, or used3D models for re-lighting. These techniques are too compu-tationally expensive for our real-time interactive applicationson normal hardware. However, to approach photorealism, theoutput video must have credible lighting and colour.

Figure 7: An extreme case of face re-colouring: left, the an-imated blended directly onto background. Right, the face re-coloured using our approach, then blended on

The output from the expression ratio image has been nor-malised in luminance, and inherits its colour distribution di-rectly from the puppet’s initial image. It therefore needs sig-nificant re-colouring in order to be able to blend well with thepuppeteer’s video stream. Various different recolouring tech-niques are implemented, such as histogram-matching in thechroma (YUV) or red-green-blue (RGB), linear shifting andmultiplication.

The most effective and least computationally expensivemethod is to work in the RGB space, assume the colour spec-tra roughly follow Gaussian distributions, and linearly scaleand shift the spectra to equalise the mean and standard de-viation for each colour component. This technique is robustto large differences in skin colour and lighting conditions be-tween the puppet and puppeteer (see Figure 7).

The colour compensation above acts in the same way oneach pixel, and cannot account for asymmetric differencesin lighting. Although local dynamic shadows and wrinklesare captured by the ERI mapping, face-wide static asymmet-rical lighting conditions (such as strong side-lighting) changelittle from the neutral reference image, and so are not well-synthesised by ERI alone. Blending the face directly withtransparency, while preserving global lighting, creates unde-sirable artefacts that reduce realism and expression transfer.Therefore, in order to preserve some of this lighting, a blurredundercoat is used (see Figure 8).

The facial region from the puppeteer image is blurred heavily,to eliminate facial features (which have a higher spatial fre-quency) but preserve face-scale lighting information (whichhas a low spatial frequency). The animated puppet face isthen blended with this undercoat, which re-lights the puppet’sface appropriately for the scene with very little computationaloverhead, without any artefacts caused by the puppeteer’s fa-cial features.

Figure 8: The facial undercoat shown above (not normallyvisible) helps to preserve local lighting conditions

4. RESULTSThe ERI system transmits dynamic shadows well. Althoughnot as strong or as well-defined in the re-synthesised video asin the source video, local shadow shapes (such as those on thenose or brow under moving light sources) are replicated. Fur-thermore, the speed of reaction to changing shadow shapes inthe re-synthesised video stream adds substantial believabilityand engagement in the case where the user is the puppeteer.

One common failure case for the system is occlusion in thepuppeteer video stream. Although the face tracker is quiterobust to occlusion, it is difficult to correct for it during facere-synthesis. Spectacles and hands in front of the face bothdisappear in the puppet face, and are blended out (see Fig-ure 9)

The software produces near-photorealistic results in a widerange of cases (see Figure 10), displayed in real time (at 15-30 fps on a 2.4 GHz PC with 8GB of RAM). The engineitself is implemented in C++, using the OpenCV library andhardware-accelerated OpenGL graphics.

Figure 9: Left: succesful affect mapping. Right: failure dueto occlusion of the face; note the glasses and hand failing todisplay realistically

A stand-alone GUI prototype for FAME has been developed,allowing for dynamic control and user/reference role switch-ing. Figure 11 shows this prototype in action with the liveuser as a puppeteer and an unlabelled image as a puppet,i.e. mode (a) operation; with windows showing puppeteer(i), puppet (ii) and re-synthesised video (iii).

Figure 10: Mapping

5. APPLICATIONS AND FURTHER WORKMode (a) operation, where the user animates a reference im-age, could be used in interactive games where the user isrepresented in the virtual space by another cracter. Interac-tive expression mapping may help the user relate to their vir-tual character, which is an important consideration for seriousgames. Furthermore, real-time emotive inference techniquessuch as vision-based computational mind-reading [9] can beused to exploit the information on facial expression present inthe face-tracking system system. This information could beuseful in the design of intelligent games; for controlling theemotive response of virtual characters in reaction to the user,and so on.

Our GUI prototype operating in mode (b), where a referencevideo animates the user’s face, has been used on videos fromBaron-Cohen’s mindreading DVD [2] (see Figure 2). Thiswas a collection of videos made by professional actors, repli-cating a set of emotions to assist children with Autism Spec-trum Disorder (ASD) to recognise these emotions from fa-cial expression and body-language. Children play throughthe videos on the interactive DVD, and choose a representa-tive expression for each video clip. Although with training,the children successfully recognised the emotions expressed

in the videoclip dataset, they were less successful at recog-nising these emotions in unseen videoclips from different ac-tors [10]. It is proposed that changing the identity of the facesin these videoclips will help the children to generalise theseexpressions across people. In this instance, the ease of addingavatars to the system (with no manual assistance) becomesuseful; being able to identify with the animated face (be it theuser’s friends, or the user themselves) could improve recog-nition, learning/recall and generalisation of these emotions.

Mode (c) (users are both puppet and puppeteer) operationcould usefully be exploited to assist with communication orempathy in multi-user gaming. With the appropriate hard-ware, such as a bidirectional augmented-reality communica-tion screen or wearable augmented-reality hardware such asGoogle Glass, users could see their (or another) face on otherreal or virtual participants. The identity and facial expres-sion of other participants from the user’s viewpoint could becontrolled dynamically, which could be useful for interfer-ing with interpersonal communication in a multi-participantgame.

It would be of particular interest to combine this engine witha generative system. For instance, with reference videos pro-viding the appropriate ERI texture mapping, the control in theshape-modification stage of expression manipulation couldbe exploited to introduce realistic variability. It would alsobe possible to extend the system with other, more sophisti-cated synthetic forms of facial manipulation at identity level:changing or varying age, race, gender and so on.

Figure 11: A graphical-interface prototype; windows for in-put video stream and expression control (i), puppet avatar (ii)and affect-mapped video output (iii)

6. CONCLUSIONSWe have presented a Facial Affect Mapping Engine, a toolfor use in interactive intelligent games where the modifica-tion, transposition or generalisation of facial expression ofemotion is important. It advances the state-of-the-art in real-time convincing, separate expression and identity mapping onnon-specialised hardware, which is of real value for interac-tive intelligent games. Both the puppet and puppeteer canbe controlled either from a reference video or from a stan-dard web-cam, giving rise to three distinct modes of opera-tion. The expressions from the puppeteer can be altered andreliably mapped to a synthesised face which maintains thepuppet’s identity. Finally, the synthesised face is graphicallyblended into either the puppet or puppeteer’s source video,with a high degree of realism. This technique for facial affectmapping has important applications in interactive intelligentgames, particularly in cases where facial expression and iden-tity need to be separated realistically. It facilitates the gener-alisation of expressed emotions, and allows a game to ’trick’users by dynamically changing the expressions and identitiesof other participants.

7. ACKNOWLEDGEMENTSThe research leading to these results has received fundingfrom the European Community’s Seventh Framework Pro-gramme (FP7/2007-2013) under grant agreement n 289021(ASC-Inclusion).

REFERENCES1. Baltrusaitis, T., Robinson, P., and Morency, L.-P.

Constrained local neural fields for robust faciallandmark detection in the wild. In InternationalConference on Computer Vision and workshops (2013).

2. Baron-Cohen, S., Golan, O., Wheelwright, S., and Hill,J. J. Mind reading: the interactive guide to emotions.

3. Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., andNayar, S. K. Face swapping: automatically replacingfaces in photographs. In ACM Transactions on Graphics(TOG), vol. 27, ACM (2008), 39.

4. Boker, S. M., Cohn, J. F., Theobald, B.-J., Matthews, I.,Brick, T. R., and Spies, J. R. Effects of damping headmovement and facial expression in dyadic conversationusing real–time facial expression tracking andsynthesized avatars. Philosophical Transactions of theRoyal Society B: Biological Sciences 364, 1535 (2009),3485–3495.

5. Castro, A. Faces.http://arturocastro.net/work/faces.html, 2012.

6. Cristinacce, D., and Cootes, T. F. Feature detection andtracking with constrained local models. In BritishMachine Vision Conference (2006).

7. Dale, K., Sunkavalli, K., Johnson, M. K., Vlasic, D.,Matusik, W., and Pfister, H. Video face replacement. InACM Transactions on Graphics (TOG), vol. 30, ACM(2011), 130.

8. Ekman, P., Friesen, W. V., and Ellsworth, P. Emotion inthe Human Face, second ed. Cambridge UniversityPress, 1982.

9. El Kaliouby, R., and Robinson, P. Generalization of avision-based computational model of mind-reading. InAffective computing and intelligent interaction.Springer, 2005, 582–589.

10. Golan, O., and Baron-Cohen, S. Systemizing empathy:Teaching adults with asperger syndrome orhigh-functioning autism to recognize complex emotionsusing interactive multimedia. Development andpsychopathology 18, 2 (2006), 591.

11. Lillard, A. Pretend play as twin earth: A social-cognitiveanalysis. Developmental Review 21, 4 (2001), 495–531.

12. Liu, Z., Shan, Y., and Zhang, Z. Expressive expressionmapping with ratio images. In Proceedings of the 28thannual conference on Computer graphics andinteractive techniques, ACM (2001), 271–276.

13. Robinson, P., and el Kaliouby, R. Computation ofemotions in man and machines. PhilosophicalTransactions of the Royal Society B: Biological Sciences364, 1535 (2009), 3441–3447.

14. Saragih, J. M., Lucey, S., and Cohn, J. F. DeformableModel Fitting by Regularized Landmark Mean-Shift.International Journal of Computer Vision 91, 2 (2011),200–215.

15. Saragih, J. M., Lucey, S., and Cohn, J. F. Real-timeavatar animation from a single image. In Automatic Face& Gesture Recognition and Workshops (FG 2011), 2011IEEE International Conference on, IEEE (2011),117–124.

16. Torresani, L., Hertzmann, A., and Bregler, C. Nonrigidstructure-from-motion: estimating shape and motionwith hierarchical priors. IEEE Transactions on PatternAnalysis and Machine Intelligence 30, 5 (May 2008),878–892.

17. Tzeng, V., Cheng, Y.-T., Liang, Y., Wang, C.-C., Chen,B.-Y., Chuang, Y.-Y., and Ouhyoung, M.3d-model-based face replacement in video.

18. Weise, T., Li, H., Van Gool, L., and Pauly, M. Face/off:Live facial puppetry. In Proceedings of the 2009 ACMSIGGRAPH/Eurographics Symposium on ComputerAnimation, ACM (2009), 7–16.