Saliency-Based Artistic Abstraction With Deep Learning and ......Shakeri, Nixon, and DiPaola: Saliency-based artistic abstraction with deep learning and regression trees with Haeberli’s

Journal of Imaging Science and Technology Rcopy 61(6) 060402-1ndash060402-9 2017ccopy Society for Imaging Science and Technology 2017

Saliency-Based Artistic AbstractionWith Deep Learning andRegression Trees

Hanieh Shakeri Michael Nixon and Steve DiPaolaSimon Fraser University 250mdash13450 102nd Avenue Surrey BC V3T 0A3

E-mail hshakerisfuca

Abstract Abstraction in art often reflects human perceptionmdashareasof an artwork that hold the observerrsquos gaze longest will generally bemore detailed while peripheral areas are abstracted just as theyare mentally abstracted by humansrsquo physiological visual processThe authorsrsquo artistic abstraction tool Salience Stylize uses DeepLearning to predict the areas in an image that the observerrsquos gazewill be drawn to which informs the system about which areas tokeep the most detail in and which to abstract most The planarabstraction is done by a Random Forest Regressor splitting theimage into large planes and adding more detailed planes as itprogresses just as an artist starts with tonally limited masses anditerates to add fine details then completed with our stroke engineThe authors evaluated the aesthetic appeal and effectiveness ofthe detail placement in the artwork produced by Salience Stylizethrough two user studies with 30 subjects ccopy 2017 Society forImaging Science and Technology[DOI 102352JImagingSciTechnol2017616060402]

INTRODUCTIONThroughout history artists have made attempts to mimichuman perception in their creations For example the Cubistart movement gave rise to works such as Jean MetzingerrsquosFemme au miroir1 and Albert Gleizesrsquo Femme au gantnoir2 where the background detail is far more abstractthan the foreground detail reflecting the Gestalt psychologyobservation of greater salience in the foreground

Zeki discusses art as an externalization of the brainrsquosinner workings bringing up mental abstraction as a solutionto the overwhelming sensory task of attending to everydetail and the limitations of the human memory system3The artistrsquos approach to abstraction Zeki claims holdsmany similarities to the physiological visual process Visuallysalient areas of the art will be detailed (either in color andtonal values texturing size or complexity of shapes) whilelsquolsquoperipheralrsquorsquo or non-salient areas will be more abstracted

The artistrsquos approach to emphasizing detail in salientareas is an iterative process as explained by well-knownportraitist John Howard Sanden in his book Portraits fromLife4 The first pass on the creation of art usually startswith a small number of large tonally limited masses Witheach iteration the artist creates smaller details and morevaried tonal values in salient areas Sanden describes this as atwo-stage process where the subject is first visually reduced

Received June 25 2017 accepted for publication Sept 18 2017 publishedonline Nov 21 2017 Associate Editor Susan Farnand1062-3701201761(6)0604029$2500

to three or four largemasses and secondly the smaller detailswithin each mass are painted while keeping the large mass inmind4

With the advent of the Artificial Intelligence subfieldof Deep Learning Neural Networks (Deep Learning) andthe ability to predict human perception to some extent thisprocess can be automated modeling the process of abstractart creation This article discusses a novel artistic abstractiontool that uses a Deep Learning visual salience predictionsystem which is trained on human eye-tracking maps ofimages of common objects Given an input image thissystem generates a predicted eye-tracking salience heatmapof the image which is then used by our tool to segmentthe image into abstraction levels While our technique isapplicable to many forms of modernist fine art paintingand abstraction in this article we concentrate our generatedoutput techniques on one formof artistic abstractionwhich isplanar abstraction most notated in the art field of AnalyticalCubism popularized by Picasso and Braque

The abstraction process itself similar to the humanartist starts with a high level of abstraction and iterativelyadds more detail to salient areas This is done with aRandomForest Regressor which splits the abstraction planesinto polygons of smaller and smaller sizes The process isrepeated at different depths or levels of detail for each ofthe abstraction levels from the generated salience lsquolsquoheatmaprsquorsquowhich are then combined into one cohesive image

The end output of our tool is an abstract art rendition ofthe input image with higher levels of planar detailing aroundareas of visual salience and large tonally simple planes in thevisually lsquolsquoperipheralrsquorsquo areas of the imageWe envision SalienceStylize mainly as an intelligent non-photorealistic rendering(NPR) technique for artists to use in the creation of digital artHowever there are other uses as well such as anonymizationAn example of this is Kadir et alrsquos work which explores theuse of artistic processing as an anonymization practice indocumentaries in order to preserve the integrity of the imageand to avoid victimizing subjects through the blurring oftheir faces5

RELATEDWORKSOur research being at the intersection of NPR cognitivescience and machine learning builds on works in each ofthese fieldsWithinNPR research a number of painterly ren-dering techniques were developed in the early 1990s starting

J Imaging Sci Technol 060402-1 Nov-Dec 2017

Shakeri Nixon and DiPaola Saliency-based artistic abstraction with deep learning and regression trees

with Haeberlirsquos pioneering work6 introducing stroke-basedpainting and Litwinowiczrsquos7 fully automated algorithmproducing paintings by using short linear paint strokestangent to Sobel edge gradients Hertzmann89 advanced thefield by using a multi-pass system of coarse-to-fine curvedb-spline strokes aligned to a layered coarse-to-fine imagedifference grid with multiple styles This was inspired by hisobservation that artists begin a painting with a first pass oflarge broad strokes and then refined the process with smallerstrokes to create detail Inspired by Hertzmannrsquos workDiPaola10 moved away from amulti-pass coarse-to-fine gridfor stroke placement choices and toward tonal masses basedon lighting and drawing types DiPaola created a color systemthat samples the tone (grayscale value) first then with thattone remaps into a color model of choice which differs fromtypical NPR color choice algorithms

A number of NPR methods specific to abstract art havebeen developed as well one example being Collomossersquos11cubist rendering technique This technique takes as inputtwo-dimensional images of a subject from different perspec-tives and after identifying the salient features in the imagescombines them in one rendering to create the illusion ofdepth and motion Finally an image rendering algorithmpass gives the piece the textural appearance of a painting

Many cognitive scientists have asserted and givenvarying degrees of evidence that visual artists exploit theproperties of the human cognitive and visual systems12ndash17For all humans researchers have shown that vision is anactive or a constructive process161819 We lsquolsquoconstructrsquorsquothe internal model based on the representation of theseexperiences13141819 This emphasizing the salient is whatZeki3 refers to as the Law of Abstraction in which theparticular is subordinated to the general so that it canrepresent many particulars Ramachandran and Hirstein20proposed eight principles in their cognitivemodel of artmostnotably the peak shift effect which is similar to the theoryof exaggerating the salient features that distinguish a givenobject of interest from other objects

An automatic system for salience adaptive paintingdriven by machine learning was presented by Collomosseand Hall21 Collomosse Gooch and others2122 described amajor problem ofmost NPR approachesmdashthey implementedlocal not global techniques and therefore restricted lsquolsquoattentionto local image analysis by independently examining smallpixel neighborhoodsrsquorsquo and these local techniques lsquolsquocan giveno real indication of salience in an imagersquorsquo23 Given that blindlocal processinganalysis especially in image-based NPR wasa major impediment to provide artistic salience in NPRhe and several other researchers looked for efficient globalsolutions212224ndash26

ProcessThe development of this tool started with the analysisof stylize a planar abstraction system created by Githubuser Alec Radford27 This system uses a regression tree-based approach specifically using the scikit-learn libraryrsquosensemble learning Random Forest Regressor28 This system

Figure 1 A picture of a woman abstracted at a single depth value

begins by simplifying the image into a single-tone polygonand as the regression treersquos depth increases (up to a terminalmax_depth value) lsquolsquochildrsquorsquo polygons are created by splittingthe lsquolsquoparentrsquorsquo into smaller more detailed polygons and variedcolors

We approached this analysis by thinking of the systemas an artificially intelligent abstract artist As we tested thissystem we realized an opportunity for improvement in thefact that it was applying a single depth value to the wholeimage (Figure 1) We noted that abstraction in art is a toolfor reducing unnecessary detail and not generally appliedequally to all areas of the work

Another significant improvement we aimed to make inthis system was to consider from a cognitive viewpoint theapproach that a human artist takes when deciding where toadd detail in an image As a result we decided to determinethe salient areas of an image using the deep convolutionalnetwork saliency prediction system (SalNet) introduced byPan et al29 SalNet was developed on Caffe and was trainedwith the SALICON (Salience in Context) dataset which iseye-tracking salience data for the MS COCO (MicrosoftCommon Objects in Context) image database Given aninput image this system generates a black and white salienceheatmap of the predicted eye-tracking data (Figure 2)

We processed the images in a vector format as this givesmore flexibility for future work built on this tool as well asthe ability to scale images easily We used the potrace svg toolto handle this conversion

METHODOur tool Salience Stylize generates planar abstract artby creating three abstraction levels in an input image byreferring to a salience heatmap of predicted eye-tracking data(Figure 3) First the input image is passed through the SalNetsystem in order to generate the saliencymap which haswhiteareas for highest saliency black areas for lowest saliency andgray tones for mid-range values The generated map has thesame dimensions as the input image and generally has bestresults for input images where there is a clearly dominantsubject such as portraits

Figure 2 The Salience Stylize art creation process

Figure 3 An input image (left) and the generated saliency heatmap (right)

Next the image is passed through the stylize tool at thefollowing depth values

1 5 (very abstract large planes)2 10 (mid-level abstraction slightly smaller planes)3 20 (detailed very small planes)

The resulting abstract renders are then converted toblack and white outlines as the potrace svg system worksonly in black and white This outlining process divides theplanes into groups of black pixels which will then be treatedas paths by potrace This grouping happens as follows

group= P|P subA Cpxpy = Cpx+1py Cpxpy = Cpxpy+1

(1)where P is a pixel A is the abstract image and C is the colorof P at x and y coordinates The resulting outline images arethen passed through the potrace program to get three vectorimages where each path is one abstracted plane

The next step is to filter the saliency areas based on thegenerated saliency map deciding which paths to keep in thearea and which to delete

high= G|Gsub svg 115leMGxGy ge 255 (2)

mid= G|Gsub svg 30leMGxGy lt 115 (3)low= G|Gsub svg 0leMGxGy lt 30 (4)

where G is a path or plane svg is the vector image of theoutlines and M is the saliency map at the x and y locationsof the paths Since the saliency map is monochrome thefull range of values is from 0 to 255 0 is fully black andnon-salient while 255 is fully white and salient

For the most detailed outlines any path that is not inthe high set defined above is removed and the same is donefor the mid-detail outlines and the mid set as well as themost abstract outline image and the low set This leaves largeabstract planes in the non-salient areas mid-sizes planes inthemid-saliency areas and small detailed planes in themostsalient areas of the map

At this stage the planes are all black andmust be coloredFor each abstraction level the color of the plane is copiedfrom the related plane in the original abstract renders Thethree levels of paths are all then appended to one array ofpaths which result in one cohesive svg image with all threeabstraction levels

The image is then passed through our NPR painterlytoolkit known as ePainterly which uses several passes to

Figure 4 The post-painterly image before (top) and after (bottom) color correction with close-ups

achieve a hand painted style including (1) natural lookinglsquolsquohandrsquorsquo distortion of the planes (2) an lsquolsquoink linersquorsquo (painteddark thin line) on some edges that is correlatedwith the natu-ral distortion and (3) lastly we use a sophisticated cognitive-and particle-system-based painterly brush stroking systemto repaint (brush stroke) all parts of the image which givetexture and detail to the planes

We do this with two modules of our ePainterly NPRsystem which was adapted from its typical use to workspecifically with the style attributes and color toning ofanalytic cubism The first takes the solid colored planes fromthe Salience Stylize system as source analyzes it and perturbsthe straight edges to be more natural looking with a slightdistortion of the planes based on two Perlin noise functions(one for each coordinate) To achieve the hand painted lookwe have experimented with lsquolsquoroughnessrsquorsquo of the Perlin noisewhich is relative transparency of the octaves the noise iscomposed of (the number of octaves is controlled by theDetails parameter) to achieve the best effectWe then achievecorrelated ink lines to the noise by producing this noiseprocess twice and merging the resultsmdashthe lines show upwhere there is a difference in the two functions which occurson some edges as seen on many cubists works by Picasso

We then repaint the whole image using our cognitive-based painting system Painterly1030 which models the

cognitive processes of artists uses algorithmic particlesystem and noise modules to generate artistic color palettesstroking and style techniques We applied this three-stageePainterly process on all our generated output

Finally the resulting image is color-corrected as the col-ors were oversaturated in order to preserve the details whenpassed through ePainterly In order to inform the choice ofcolor palette we used the pyCaffe-based implementation ofstyle transfer31 with one of our output images as the contentimage and awork of professional cubist art as the style imageAs this process added noise or at times distorted the imagewe did not use this in our final work but instead learnedfrom its output and shifted the tones of our image toward theoutput of the style transfer This resulted in a more mutedcolor palette (Figure 4)

Study 1 Evaluation of Aesthetic AppealMethodLeder et al32 introduce a model for aesthetic appreciationof art in which previous experience as well as the styleand content of the artwork are factors in aesthetic likingand aesthetic emotion Style and content were judged bytheir interestingness and emotional arousal We evaluatedthese factors of aesthetic appeal of the artwork producedby Salience Stylize through a user study on the Amazon

Mechanical Turk platform using a combination of jsPsych33and Psiturk34 to facilitate the study Psiturk provides ad-serving for psychological studies running on MechanicalTurk and also the backend server to run the ads andhas beengenerally found to be a reliable mechanism for conductingexperiments35 JsPsych is a JavaScript-based library forproviding a wide range of psychological instruments formeasuring participant response while controlling the flowof the participantsrsquo stimuli and responses

One Human Intelligence Task (HIT) with 30 assign-ments was loaded onto AmazonMechanical Turk 30 uniqueworkers completed both trials We denied the use of mobileor tablet devices for thiswork in order tomaximize the imagequality and size when viewed by participants

InstrumentsWe designed two questionnaires The first was presented toeach participant before the study and asked the participantto rate their expertise with art in general as well as withanalytical cubist art The second questionnaire included aweb-based interactive form that presented a total of 34paintings one at a time to the participant along witha set of questions and Likert scales next to the imageto be answered for each painting Before starting thequestionnaire the participants were asked to consider eachimage independently without reference to previous imagesand to evaluate them as works of fine art These questionsremained the same for every painting

(1) How well do you think this artwork depicts the essenceof the subject (On a scale of 1ndash9)

(2) How pleasing is the style of this painting (On a scale of1ndash9)

(3) How interesting is this painting (On a scale of 1ndash9)(4) Rate this painting out of 5 as a work of art (in terms of

style content and sophistication)

The images presented to the participants included twovariations of four paintings produced by Salience Stylize(including the saliency-based abstraction and either onlyhigh detail only low detail or randomized abstraction) and19 works of professional and amateur cubist art (Figure 5)Three of the professional paintings and three of the amateurpieces were repeated in order to avoid bias caused bysimilarity between variations of Salience Stylize output Thisartworkwas collected as the top results from aGoogle Imagessearch for lsquolsquoanalytical cubismrsquorsquo that is from this search the9 to 10 of first images that were by professional artists andthe first top 9 to 10 images that were from non-professionalartists were used The first 10 professional paintings ended upbeing 5 from Picasso and 5 from Braque the masters of thisfield The total imageset was shuffled before being presentedto the user

ResultsOf the 30 participants one was removed as their completiontime of five minutes was much shorter than expected for the

Figure 5 Four variations of different imagesmdashlow detail saliency-basedrandom abstraction and high detail (left to right top to bottom)

number of questions Of the 29 remaining 16 identified asmales and 13 as females They were aged between 21 and 64(M = 3448 SD= 1045) 10 had very little or no familiaritywith art in general 16 had some previous familiarity and 3considered themselves experts in art

In order to assess how the different kinds of artworkcompared to each other we examined each aggregate cate-gory our saliency-based abstraction our non-saliency-basedabstraction the amateur cubist set and professional cubistset We wanted to determine whether the saliency-basedmodel produced work that was more similar to professionalcubist or amateur cubist work For all the artwork that wasrated we asked four questions rating its Essence StyleInterest and Artistic Quality (Figure 6) The comparisonswe were mainly interested in were saliency versus amateursaliency versus professional and saliency versus non-saliency To compare these means we used homoscedastic2-sample t-tests testing at the 005 significance level Wefirst successfully conducted an f -test to ensure that theassumption of equal variances was acceptable

First we wanted to establish that the amateur andprofessional artworks were perceived differently This wouldbe the baseline as it would show that the sets of confederateartwork used alongside our generative artwork were distinctWhen comparing the amateur versus professional artworksrsquoEssence ratings there was a significant difference betweentheir means t(723) = 196 p = 0036 When comparingthe amateur versus professional artworksrsquo Style ratings therewas a significant difference between their means t(723) =

Figure 6 Performance of each art category on the four questions (therating category was out of 5)

196 p lt 00001 When comparing the amateur versusprofessional artworksrsquo Interest ratings there was a significantdifference between their means t(723)= 196 p= 000037When comparing the amateur versus professional artworksrsquoArtistic Quality there was a significant difference betweentheir means t(723) = 196 p lt 00001 This seems tosupport our hypothesis that our amateur and professionalpools of artwork are visually distinct

Next we wanted to compare our saliency category ofgenerative artwork to the set of amateur artworks Thiswouldallow us to determine if they were perceived similarly tothat kind of artwork When comparing the saliency versusamateur artworks there was a significant difference betweentheir means of all four questions Essence ratings (t(462)=

197 p= 00076) Style ratings (t(462)= 197 p= 00037)Interest ratings (t(462) = 197 p = 00075) and Artisticratings (t(462) = 197 p = 00016) This supports thehypothesis that our saliency category of generative artworkis visually distinct from the amateur artworks

Next we wanted to compare the saliency category ofgenerative artworks to the set of professional (ie Picassoand Braque) artworks When comparing saliency versusprofessional artworksrsquo ratings we did not find significantdifferences between themeans of any of the four ratings Thismeans that we can support a hypothesis that our generatedsaliency artworks are not separate from the professionalartworks To our participants as a category these worksreceived similar ratings

Lastly we wanted to compare our saliency and non-saliency generative artworks This would let us see whetherthey were distinct categories of work When comparing thesaliency versus non-saliency artworksrsquo Essence ratings as wellas the Artistic Quality ratings there was no significant dif-ference between their means When comparing the saliencyversus non-saliency artworksrsquo Style ratings (t(260)= 211p= 0036) and Interest ratings (t(260)= 211 p= 00085)there was a significant difference between their means Thismeans that in two of the ratings (Style and Interest) wefind support for these categories being different howeverfor two of the ratings (Essence and Artistic Quality) wecannot support that hypothesis There seems to be somedistinctiveness to these two categories of artwork

Table I Number of participants who selected each version

Image High detail Saliency-based Low detail Random detail

Image 1 24 0 3 2Image 2 16 7 3 3Image 3 16 7 3 3Image 4 11 8 6 4Image 5 15 8 3 3Image 6 12 8 5 4Image 7 14 5 2 8Image 8 13 5 4 7

Study 2 Evaluation of Detail PlacementMethodThis study was presented as the second part of the Study1 with the same participants The purpose of this part wasto ascertain that the placement of highest detail on themost salient areas of an image creates the most aestheticallypleasing artwork We tested this by presenting eight outputpaintings (four ofwhichwere shown in Study 1) but this timewith four variations of each painting

InstrumentsFor each painting we juxtaposed the following variations indifferent orders

1 Only high detail2 Only low detail3 Three levels of detail most detailed at high saliency areas4 Random detail generated with a false saliency map of

Perlin noise

On each page the user was asked to select the variationthat created the most aesthetically pleasing work of abstractart (Figure 7)

ResultsWhen analyzing the responses of all participants the highdetail images were always selected more often than theother variations (Table I) However when limiting resultsto only those who identified as art experts high detail andsaliency-based images were each selected 7 times in totalwhile low and random detail images were each selected 5times (Table II)

When limiting the results to only male participantssaliency-based images were selected a total of 29 timeswhile high detail images were selected 68 times which is aratio of approximately 043ndash1 When limiting to only femaleparticipants saliency-based images were selected a total of 19times and high detail images were selected 53 times whichis a ratio of approximately 036ndash1

DISCUSSIONThe results of Study 1 confirm that the quality and aestheticappeal of the art produced by Salience Stylize are perceived

Figure 7 Four variations of abstraction including 1mdashrandom abstraction 2 mdashsaliency-based abstraction 3mdashlow detail and 4mdashhigh detail

Table II Number of expert participants who selected each version

as better than both non-saliency-based abstraction andamateur cubist artwork scoring relatively close to theprofessional cubist artwork for each of the questions Forevery question there was a significant difference between themean responses for saliency-based abstraction and amateurcubist artwork while there was no significant differencebetween the mean responses for saliency-based abstractionand professional cubist artwork This provides support forthe quality and aesthetic appeal of our art pieces being high

The results of Study 2 did not support our hypothesisthat saliency-based detail placement would produce themostappealing art pieces as participants were overall more likely

to select the entirely high detail images This finding isin line with previous research36 showing that for viewerswith no expertise in cubist art the preference is for thoseimages in which objects are most recognizable Hekkertand Leder37 claim that lsquolsquowe like to look at patterns thatallow us to see relationships or create orderrsquorsquo However thispreference for high detail was not as strongwhen consideringonly the participants with an expertise in art as an equalnumber selected the saliency-based images This discrepancybetween expert and non-expert participantsrsquo responses foraesthetic appeal evaluation has been explored previouslyby researchers such as Lebreton et al38 Male and femaleparticipants did not have significantly different preferencesThis suggests that when creating generative artwork it is stillcrucial to keep onersquos audience inmind along with the impactthis has on evaluation

Future DirectionsWhile Salience Stylize mimics an artistrsquos abstraction processby simplifying the areas of least visual salience an argumentcan be made that this does not fully reflect the intricaciesof the artistrsquos design decisions Rather than being a way ofrecognizing where an observerrsquos gaze will fall and increasingthe detail there artistic abstraction and detailing sometimesserves as a tool for an artist to intentionally guide theobserverrsquos gaze to another area of the artwork

An eye-tracking study by DiPaola et al17 providessupport for this as they found that the gaze of participantswas drawn to areas of finer textural detail in Rembrandt-styleportraits and suggested that lsquolsquoportrait artists perhaps even asearly as Rembrandt guided the viewerrsquos gaze through theirselection of textural detailrsquorsquo

This variation in the abstraction process is not addressedin our work but this provides an interesting opportunityfor improvement in future works In order to create a truedigital abstract artist it is necessary to accommodate theoption to create abstract art that is not influenced by saliencybut rather acts as an influencer of saliency Perhaps thiseffect could even be approximated through Deep Learningby training a Neural Network on variations in detailing inartwork and using that to predict the areas that an artistwould select for finer textural detailing This could replacethe saliency prediction system (SalNet) which is currentlyused in our work Alternately the use of a Neural Networkfor the generation of saliency maps could be replaced orenhanced by a human user In such a setup the user wouldhave greater creative influence over the output and would beable to select the areas of the artwork in which greater detailis required

Another possible solution to this is to model thetraditional art-studio setup of a painter observing a subjectBy interacting with the subject a painter will gain a betterunderstanding of their personality or lsquolsquoessencersquorsquo whichthey can then reflect in the portrait This process couldbe automated by capturing on camera the unique facialexpressions and gestures of an individual over a short spanof time before taking their photograph The detail placementof the portrait could then be influenced by this processemphasizing the subjectrsquos most expressive facial features fora truly personalized portrait

Currently Salience Stylize is limited to the style oftraditional Analytical Cubism however in future works thiscan be generalized to other art styles as well For examplewe have experimented with the use of Googlersquos Deep Dream(an image processing approach for visualizingDeep LearningNeural Networks)39 in conjunction with saliency maps forstylizing the images which creates a less traditional look Inthis variation we also explored the use of different detailingtechniques in salient areas such as the number of lsquolsquoiterationsrsquorsquoin Deep Dream which can create more saturated colors andsharper edge lines in salient areas

CONCLUSIONWe propose a novel method of abstract art creation basedon predicted human saliency of images using Deep Learningand a Random Forest Regressor Abstraction in art oftenreflects human perception and we aimed to reflect thatby omitting details in the peripheral image areas andemphasizing details in the areas that a viewer would bemost likely to look Similar to a human artistrsquos methodof starting with a few large tonally limited masses andadding increasingly refined detail with each iteration ourtool Salience Stylize starts by splitting the image into large

planes and adding more detailed planes as it progressesWe evaluated the aesthetic appeal and detail placementin the artwork produced by Salience Stylize through twouser studies with 30 subjects and found that our artworkwas considered by participants as having a quality closeto professional cubist artwork however we found that ourartwork did not perform as well as images of entirelyhigh detail when assessing the effectiveness of the detailplacement Finally we recommend further developing thismethod in order to address the need for artist-drivensaliency where intelligent decisions about detail placementguide the viewerrsquos perception rather thanmodeling the detailplacement on a prediction of viewersrsquo gaze

ACKNOWLEDGMENTThe authors thank Suk Kyoung Choi Graeme McCaigVanessa Utz and Ulysses Bernardet for their suggestions andfeedback in the process of writing this article and validatingtheir results This research was supported by a KEY Big DataUSRA and SSHRC

REFERENCES1 A Gleizes Femme Au Gant Noir (Woman with Black Glove) oil oncanvas 126 cmtimes 100 cm 1920 Retrieved from httpsenwikipediaorgwikiFileAlbert_Gleizes_1920_Femme_au_gant_noir_(Woman_with_Black_Glove)_oil_on_canvas_126_x_100_cm_Private_collectionjpg

2 J Metzinger Femme Au Miroir (Femme agrave Sa Toilette Lady at HerDressing Table) oil on canvas 924 cmtimes 651 cm 1916 Retrieved fromhttpsenwikipediaorgwikiFileJean_Metzinger_1916_Femme_au_miroir_(Femme_C3A0_sa_toilette_Lady_at_her_Dressing_Table)_published_in_The_Sun_New_York_28_April_1918jpg

3 S Zeki lsquolsquoArtistic creativity and the brainrsquorsquo Science 293 51ndash52 (2001)4 J H Sanden Portraits from Life in 29 Steps (North Light Books Ohio1999)

5 A Kadir O Yalcin K Hennessy and S DiPaola Proc ElectronicVisualisation and the Arts (British Computer Society London 2017) p 8

6 P Haeberli Paint by Numbers Abstract Image Representations (ACMSIGGRAPH Computer Graphics New York 1990) Vol 24 pp 207ndash214

7 P Litwinowicz lsquolsquoProcessing images and video for an impressionist effectrsquorsquoProc 24th Annual Conf on Computer Graphics and Interactive Techniques(ACMPressAddison-Wesley Publishing Co NewYork NY USA 1997)pp 407ndash414 doi httpdxdoiorg101145258734258893

8 AHertzmann lsquolsquoPainterly renderingwith curved brush strokes ofmultiplesizesrsquorsquo Proc 25th Annual Conf on Computer Graphics and InteractiveTechniques (ACM New York NY USA 1998) pp 453ndash460 doi httpdxdoiorg101145280814280951

9 A P Hertzmann Algorithms for Rendering in Artistic Styles (New YorkUniversity Graduate School of Arts and Science New York 2001)

10 S Dipaola lsquolsquoExploring the cognitive correlates of artistic practiceusing a parameterized non-photorealistic toolkitrsquorsquo (University of BritishColumbia Vancouver Canada 2013)

11 J P Collomosse and P M Hall lsquolsquoCubist style rendering from pho-tographsrsquorsquo IEEE Trans Vis Comput Graphics 9 443ndash453 (2003)

12 P Cavanagh lsquolsquoThe artist as neuroscientistrsquorsquo Nature 434 301ndash307 (2005)13 S Grossberg lsquolsquoThe art of seeing and paintingrsquorsquo Spatial Vis 21 463ndash486

(2008)14 P Mamassian lsquolsquoAmbiguities and conventions in the perception of visual

artrsquorsquo Vis Res 48 2143ndash2153 (2008)15 W P Seeley lsquolsquoWhat is the cognitive neuroscience of art andwhy should

we carersquorsquo Am Soc Aesthetics 31 1ndash4 (2011)16 S Zeki Inner Vision An Exploration of Art and the Brain (Oxford

University Press USA 2000)

17 S DiPaola C Riebe and J T Enns lsquolsquoRembrandtrsquos textural agency Ashared perspective in visual art and sciencersquorsquo Leonardo 43 145ndash151(2010)

18 J T Enns The Thinking Eye the Seeing Brain Explorations in VisualCognition 1st ed (W W Norton amp Company New York 2004)

19 P Cavanagh lsquolsquoVisual cognitionrsquorsquo Vis Res 51 1538ndash1551 (2011)20 V S Ramachandran andW Hirstein lsquolsquoThe science of art A neurological

theory of aesthetic experiencersquorsquo J Consciousness Studies 6 15ndash51 (1999)21 J P Collomosse andPMHall lsquolsquoPainterly rendering using image saliencersquorsquo

Proc 20th Eurographics UK Conf (DeMontfort University Leicester UK2002) pp 122ndash128

22 BGooch andAGoochNon-photorealistic Rendering (NatickMAUSAAK Peters Ltd 2001)

23 J P Collomosse lsquolsquoHigher level techniques for the artistic rendering ofimages and videosrsquorsquo Doctoral thesis (University of Bath Bath UK 2004)

24 B Gooch G Coombe and P Shirley lsquolsquoArtistic vision painterly renderingusing computer vision techniquesrsquorsquo NPAR rsquo02 Proc 2nd Intrsquol Symposiumon Non-photorealistic Animation and Rendering (New York USA 2002)pp 83ndashff

25 H Huang T-N Fu and C F Li lsquolsquoPainterly rendering with content-dependent natural paint strokesrsquorsquo Vis Comput 27 861ndash871 (2011)

26 M Kagaya W Brendel Q Deng T Kesterson S Todorovic P J NeillandE Zhang lsquolsquoVideo paintingwith space-time-varying style parametersrsquorsquoIEEE Trans Vis Comput Graphics 17 74ndash87 (2011)

27 A Radford stylize Regressor based image stylization 2017 Retrievedfrom httpsgithubcomNewmustylize

28 F Pedregosa G Varoquaux A Gramfort V Michel B Thirion O GriselM Blondel P Prettenhofer R Weiss V Dubourg J Vanderplas A PassosD Cournapeau M Brucher M Perrot and Eacute Duchesnay lsquolsquoScikit-learnMachine learning in pythonrsquorsquo J Mach Learn Res 12 2825ndash2830 (2011)

29 J Pan E Sayrol X Giro-i-Nieto K McGuinness and N E OrsquoConnorlsquolsquoShallow and deep convolutional networks for saliency predictionrsquorsquoProc IEEE Conf on Computer Vision and Pattern Recognition (IEEEPiscataway NJ 2016) pp 598ndash606

30 S DiPaola lsquolsquoExploring a parameterised portrait painting spacersquorsquo Int JArts Technol 2 82ndash93 (2009)

31 L A Gatys A S Ecker and M Bethge lsquolsquoA neural algorithm of artisticstylersquorsquo arXiv150806576 [cs q-bio] (Sept 2015)

32 H Leder B Belke A Oeberst and D Augustin lsquolsquoA model of aestheticappreciation and aesthetic judgmentsrsquorsquo Br J Psychol 95 489ndash508 (2004)

33 J R De Leeuw lsquolsquojsPsych A JavaScript library for creating behavioralexperiments in a Web browserrsquorsquo Behav Res Methods 47 1ndash12 (2015)

34 J V McDonnell J B Martin D B Markant A Coenen A S Rich andT M Gureckis psiTurk (New York University New York 2012)

35 M J Crump J V McDonnell and T M Gureckis lsquolsquoEvaluating AmazonrsquosMechanical Turk as a tool for experimental behavioral researchrsquorsquo PloS one8 e57410 (2013)

36 C Muth R Pepperell and C-C Carbon lsquolsquoGive me Gestalt Preferencefor cubist artworks revealing high detectability of objectsrsquorsquo Leonardo 46488ndash489 (2013)

37 P Hekkert and H Leder lsquolsquoProduct aestheticsrsquorsquo Product Experience(Elsevier Science Direct USA 2008) pp 259ndash285

38 P Lebreton A Raake and M Barkowsky lsquolsquoEvaluation of aestheticappeal with regard of users knowledgersquorsquo ISampT Electronic Imaging HumanVision and Electronic Imaging 2016 (ISampT Springfield VA 2016)pp HVEI-1191-HVEI-1196

39 A Mordvintsev C Olah andM Tyka 2015 Inceptionism Going deeperinto neural networks URL httpsresearchgoogleblogcom201506inceptionism-going-deeper-into-neuralhtml