1

Fake Impressionist Paintings for Images and VideoPatrick Gregory Callahan

pgcallah@andrew.cmu.edu

Department of Materials Science and EngineeringCarnegie Mellon University

May 7, 2010

Abstract—A technique for processing images and videoto create an impressionist effect has been implemented inthe Interactive Data Language (IDL). It is based on thetechnique described in Litwinowicz [1]. The technique usesbrush strokes placed randomly across an image. The brushstrokes have a user-defined length and width, and colorwhich is the mean color of a neighborhood of pixels aroundits center. The orientation of the brush stroke can be aconstant defined by the user, or the orientation can be de-termined by image gradient. Frame-to-frame (temporal)coherence between consecutive frames in a video can beachieved using the optical flow between each frame. Insteadof generating new brush strokes each frame, the centers ofthe brush strokes are moved according to the optical flow.Automated image segmentation has also been used to sep-arate images into regions which can then be rendered dif-ferently as the user prefers, for example background andforeground can have different brush stroke sizes.

Index Terms—Non-Photorealistic Rendering, OpticalFlow, Impressionist Painting, Abstract Images

I. INTRODUCTION

T HE impressionist style of painting was developed inthe nineteenth century. It was a new and abstract

representation of reality that “sought subjects in everydaylife and in the fleeting impressions of a scene [2].” Thework presented here attempts to use real images to pro-duce images that are similar to impressionist paintings.This is accomplished by applying brush strokes in the im-age in a controlled way based on the data from the originalimage. This can remove some of the higher frequency in-formation producing an abstract effect. The inspiration forthis work is the paper by Litwinowicz [1].

The impressionist effect can also be applied to videoswith frame-to-frame coherence by computing the opticalflow between frames in a video. The optical flow can thenbe used to move brush strokes from the first frame produc-ing the second frame.

Individual images can be segmented into different re-gions. These different regions can be rendered differentlyaccording to how the user defines them. Different colors,intensities, or brush strokes can be assigned to differentregions to alter the effect on the rendered image. This is

beneficial since real paintings use various brush strokes indifferent regions of the painting.

II. BACKGROUND

In 1990, Haberli [3] detailed several methods for pro-ducing abstract, impressionistic paintings from computergenerated or photographic images. The primary goal ofthe work of Litwinowicz was to produce temporally co-herent videos with an impressionist look by modifying theapproach of Haberli [1]. For a more general discussionon producing paintings from real images and other non-photorealistic rendering techniques, the reader is directedto [4].

III. METHODS AND DISCUSSION

A. Brush Stroke Generation

The paintings generated by the algorithm are composedof a number of brush strokes. The brush strokes have alength L, a radius R, and an orientation given by the angleθ. The center of the brush stroke, (cx,cy), is determined,then the mean color in a neighborhood around the center isfound. The work discussed here uses a neighborhood sizeof 4 pixels. A random list of pixel indices in the image isgenerated, the number depending on L, R, and the size ofthe image. This random list is used to choose the brushstroke centers and the order in which they are rendered.The pixel indices of the entire brush stroke are found bygenerating a rectangle at (cx,cy) with L, R, and θ. Thenthe mean color in the neighborhood surrounding (cx,cy)is assigned to each pixel inside the brush stroke.

B. Random Perturbations

Random perturbations are added to brush strokes giv-ing the images a more natural look after the technique ofLitwinowicz [1]. The red, blue, and green channel of eachbrush stroke had a random value, ∆r, ∆g, and ∆b, inthe range [-8, 8] (of image intensity [0, 255]) added toit. Litwinowicz used values in the range [-15, 15], but Ibelieve this relatively large magnitude leads to an over-processed look in the images so I used the smaller mag-nitude. The brush stroke is scaled by an intensity factor

2

Fig. 1. An original image which is taken from the Prokudin-Gorskii collection[5].

∆intensity in the range [0.92, 1.08], which is again lessthan Litwinowicz used for the same reason. After theserandom perturbations are added, the image intnsity val-ues are kept within the bounds of the byte scale, [0,255].The orientation is also perturbed by ∆θ. Figure 1 showsan original image taken from the Prokudin-Gorskii col-lection [5]. Figure 2 shows an impressionist effect ap-plied ot the image, using brush strokes having L = 10,R = 4, θ= 45◦, and including ∆r, ∆g, ∆b, ∆θ, and∆intensity.

While random perturbations can create a nice effect,their magnitudes should be controlled carefully to producethe desired effect. A more photo-realistic looking paintingwould require smaller magnitudes than I have used here.In my testing, it seemed that outdoor images look betterwith smaller perturbations while indoor pictures look al-right with larger magnitudes, however, other users maydisagree.

C. Brush Stroke Clipping

Comparing Figure 1 to Figure 2, it is apparent that thepainting algorithm does not preserve edges. The hori-zon line, the river bank, and the road, among other thingshave become quite jagged with the application of the al-gorithm, and this becomes more pronounced with largerbrush strokes. In order to maintain relatively smooth lines,edges can be used to clip brush strokes. An edge mapis generated from using the Canny Operator [6]. Prior toapplication of the brush strokes, the modified algorithmchecks whether or not it intersects any edges. When a

Fig. 2. The impressionist version of the original image withbrush strokes having L = 10, R = 4, θ= 45◦, and including∆r, ∆g, ∆b, ∆θ, and ∆intensity.

brush stroke intersects with an edge, it is divided into sep-arate regions. The region which includes (cx,cy) is thenthe only region of that brush stroke which is drawn. Themethod is illustrated in Figure 3. Figure 4 shows the effectof this method on a processed image.

The issue with the brush clipping technique is that itrequires the correct edge map to produce good results.The edge map depends on the type of edge detector used,thresholding of the edge map, and whether the image issmoothed before the edge map is generated. These issuesshould be taken into consideration for each set of imagesor video that is going to be processed to ensure that theedges are not too smooth or jagged, or too few or many.

D. Brush Stroke Orientation

Until this point in the discussion, θ was kept constantbesides random perturbations. Although the brush strokeclipping discussed in the previous section reduces theamount of aliasing along edges, it makes sense to alignbrush strokes along the direction of edges, or the gradientnormal direction. So we make the orientation equal to theedge direction, which is given by

θ = arctan(

gx

gy

)+ 90◦ (1)

where gi = ∂Is/∂i, the gradient of the gaussian blurredimage Is in the ith direction. The orientation at each pointin the image is calculated, so the orientation at the center

FAKE IMPRESSIONIST PAINTINGS FOR IMAGES AND VIDEO 3

Fig. 3. The edge map of the original image generated usinga Canny edge detector. A brush stroke that intersects an edgeis drawn and separated into different regions, and only the re-gion which is at (cx,cy) is drawn. This demonstrates the brushstroke clipping method.

Fig. 4. The effect of the brush stroke clipping method on aprocessed image is shown.

Fig. 5. The effect of using the gradient normal to determine theorientation.

of brush strokes, (cx,cy), is used to generate the impres-sionist images. Figure 5 shows an image produced usingthis method to determine the orientation θ of the brushstrokes.

Using the gradient normal direction to determine thebrush stroke orientation produces a better image in thisauthor’s opinion. Comparing Figure 4 and Figure 5 itis apparent that this automated orientation determinationmethod has allowed the brush strokes to be oriented in thedirection of the edges, especially at the horizon and theriverbank. Obviously a painter would not always followthis convention, and may prefer to sometimes paint brushstrokes in directions normal to the direction, so there maybe some additional logic that could be added to generateimages that look more hand-crafted.

E. Frame-to-Frame Coherence

It is possible to apply an impressionist effect to a videoby generating new brush strokes for every frame. How-ever, there will be no frame-to-frame coherence if thisapproach is taken. Litwinowicz [1] proposed to use theoptical flow between images to move the center of brushstrokes instead of generating new brush strokes for eachframe. So in the proposed method, the first frame is gen-erated using the methods described above. The next frameuses the same brush strokes but moves them based on theoptical flow.

To calculate the optical flow, the equation

4

0 =(∂I∂x

)Vx +

(∂I∂y

)Vy +

(∂I∂t

)(2)

must be solved for Vx and Vy the velocities in thex and y directions, respectively. First, the images areblurred using a gaussian filter, then the Lucas-Kanade [7]method is used to solve Equation 2 for Vx and Vy. TheLucas-Kanade method involves finding the optical flow ateach point by finding the gradients in a neighborhood sur-rounding the point, thenthe over-constrained system canbe solved using a method such as least squares. I used aneighborhood size of 4 pixels.

Two frames from a video are shown in Figure 6(a)-(b).The optical flow between the two images are shown in Fig-ure 6(c)-(d). The impressionist versions of the two framesare shown in Figure 6(e)-(f). The second frame was gen-erated with the same brush strokes as the first frame withthe centers of the brush strokes moved by the optical flow.

There are problems associated with this method. Sincenew brush strokes are not being generated, if brush strokesare moved to far, there will be blank regions in the image.Brush strokes can be moved past the dimensions of theimage, or many brush strokes can be moved to a smallarea, making the image cluttered and causing many of thestrokes to be covered. There are solutions to these prob-lems, but they may create other issues. Litwinowicz addedand subtracted brush strokes depending on how close orfar apart they were. The method works to keep the imagesfrom getting blank spaces, but it also adds or subtracts newbrush strokes, which can spoil the frame to frame coher-ence. The method also does not do well with changingbackgrounds, since new brush strokes must be added asthe background changes in order for the images to lookright.

F. Region Extraction

In order to differentiate between the regions ofinterest in an image, some sort of segmentationmust be performed. I decided to use Expectation-Maximization/Maximization of the Posterior Marginals(EM/MPM) algorithm described in [8], which segmentsan image by attempting to minimize the number of mis-classified pixels. This algorithm was chosen because itis a fully-automated segmentation algorithm which re-quires little user input. The problem with the algorithmthat the number of classes is user-defined and assumed tobe known prior to segmentation. Increasing the numberof classes increases the processing time of the algorithm.To overcome this, I use a small number of region labels(around 4). Although there may be many separate regionsin the image, as long as they are not contiguous they can

(a) The first frame of a video (b) The second frame of a video

(c) Optical flow in the x direction (d) Optical flow in the y direction

(e) The impressionist version of the first frame.

(f) The second frame of the video produced using the same brushstrokes after moving them based on the optical flow.

Fig. 6. (a)-(b) Two frames from a video, (c)-(d) the optical flowin the x and y directions, and (e)-(f) the impressionist paintingsgenerated from them with frame-to-frame coherence.

FAKE IMPRESSIONIST PAINTINGS FOR IMAGES AND VIDEO 5

be separated. After segmentation by the EM/MPM algo-rithm, the regions can be separated by using a Sobel edgedetector to find edges between different regions. Then allnon-contiguous regions are relabeled in order to increasethe number of region labels. Figure 7(a) shows an imagewhich has been segmented and labeled with this method.Using this method increased the number of different re-gion labels from 4 for the EM/MPM segmented image to152 in the example image shown in Figure 7(a).

Once the image has been segmented, the user needs todefine the regions of interest. This is done using a user in-terface in IDL by clicking anywhere in each region theywant to choose. A variety of things can be done oncethe regions of interest have been chosen. One can ren-der the regions of interest in color, and the other regionsin gray scale, or render the foreground more brightly thanthe background. An example showing the regions of in-terest (in this case the sky and the river were chosen) incolor and the other regions in gray scale is shown in 7(b).An example showing the regions of interest rendered morebrightly than the other regions is shown in 7(c). This is agood way to produce effects that significantly change theappearance of the images with little user input.

Painters often use different styles of brush strokes indifferent regions of a painting [9]. For example, theymay use larger brush strokes for the background of animages and finer, more detailed brush strokes in the fore-ground. Region extraction was also used to separate theforeground from the background in order to test the effectof rendering brush strokes of different sizes in differentregions. After the background and foreground were cho-sen, the impressionist algorithm was run with larger brushsizes for the background, and smaller sizes for the fore-ground. The background was then gaussian blurred, andthe two were combined. The result is shown in Figure 8.There are other methods possible for changing the size ofbrush strokes, but this is a promising method for control-ling brush strokes depending on the region.

IV. FUTURE WORK AND CONCLUSIONS

There are many possible additions or changes that couldbe made to the methods described here. The shape of thebrush stroke shape could be modified, although it couldmake brush stroke orientation unnecessary depending onthe brush stroke shape. Brush textures would also be anice addition. Brush strokes in real paintings are typicallynot a uniform shape or color, so variations in colors orshape depending on the location in the stroke, length ofthe stroke, or other variables could be added for a niceeffect.

As discussed, there are problems with the method

(a) The result of the segmentation and region labelingmethod.

(b) The segmented image has been used to generate a col-ored sky and river with other regions in gray scale.

(c) The segmented image has been used to render the skyand river with normal intensity and the rest of the imagewith lower intensity.

Fig. 7. (a) An image that has been segmented, but foregroundand background have not been chosen, and (b) the regions ofthe sky and river have been chosen .

6

Fig. 8. The effect of using large brush strokes in the backgroundand smaller brush strokes in the background. The sky and riverwere chosen as the background.

used to generate frame-to-frame coherence. While theworkarounds can help, they introduce problems as well.For a video with a static background I would propose cal-culating the mean background and using it to generate abase frame, to which brush strokes could be added or sub-tracted where there are deviations from the mean back-ground. Also, for a video with changing backgrounds, analgorithm detecting scene changes could be used to gener-ate frames with new sets of brushstrokes.

The segmented images might be able to be used to gen-erate other kinds of non-photorealistic rendering, such ascel shaded images. By leaving the edges in the image andusing the average color in each region, a nice effect mightbe produced.

There are many possible ways to produce images thatlook like paintings. The work discussed here was usedto generate paintings from real images and videos withframe-to-frame coherence. The method for frame-to-frame coherence needs additional work to behave cor-rectly with videos that have changing backgrounds. Im-ages have also been segmented in order to render differentregions with different color, intensity, or brush strokes.

ACKNOWLEDGEMENTS

Thanks to Professor Efros for doing a great job of com-bining art and science in Computational Photography andmaking the course interesting and fun. Also, thanks toLaura Trutoiu and Alvaro Collet for their help throughoutthe course.

REFERENCES

[1] Peter Litwinowicz. Processing images and video for an impres-sionist effect. SIGGRAPH Conference Proceedings, 1997.

[2] Diana Newell. The Impressionists. Ivy Press, 2008.[3] Paul Haberli. Paint by numbers: Abstract image representations.

SIGGRAPH Conference Proceedings, 1990.[4] Thomas Strothotte and Stefan Schlechtweg. Non-Photorealistric

Computer Graphics: Modeling, Rendering, and Animation. Mor-gan Kauffmann Publishers, 2002.

[5] Library of Congress. The Empire That Was Russia:The Prokudin-Gorskii Photographic Record Recreated.http://www.loc.gov/exhibits/empire/, February 2010.

[6] Wesley E. Snyder and Hairong Qi. Machine Vision. CambridgeUniversity Press, 2004.

[7] Bruce D. Lucas and Takeo Kanade. An iterative image registra-tion technique with an application to stereo vision. Proceedings ofImaging Understanding Workshop, 1981.

[8] Mary L. Comer and Edward J. Delp. The em/mpm algorithm forsegmentation of textured images: Analysis and further experimen-tal results. IEEE Transactions On Image Processing, 9(10):1731,2000.

[9] Michio Shiraishi and Yasushi Yamaguchi. An algorithm for auto-matic painterly rendering based on local source image approxima-tion. NPAR 2000: First International Symposium on Non Photo-realistic Animation and Rendering, 2000.