Collaborative Personalization of Image Enhancement

Juan C. CaicedoUniversidad Nacional

Bogota, Colombiajccaicedoru@unal.edu.co

Ashish KapoorMicrosoft Research

Redmond, WA. USAakapoor@microsoft.com

Sing Bing KangMicrosoft Research

Redmond, WA. USAsbkang@microsoft.com

Abstract

While most existing enhancement tools for photographshave universal auto-enhancement functionality, recent re-search [8] shows that users can have personalized prefer-ences. In this paper, we explore whether such personal-ized preferences in image enhancement tend to cluster andwhether users can be grouped according to such prefer-ences. To this end, we analyze a comprehensive data setof image enhancements collected from 336 users via Ama-zon Mechanical Turk. We find that such clusters do existand can be used to derive methods to learn statistical pref-erence models from a group of users. We also present aprobabilistic framework that exploits the ideas behind col-laborative filtering to automatically enhance novel imagesfor new users. Experiments show that inferring clusters inimage enhancement preferences results in better predictionof image enhancement preferences and outperforms genericauto-correction tools.

1. IntroductionDespite advances in digital photography that improve

camera functionality taking good quality pictures remain achallenge for casual photographers. Problems include in-correct camera settings and poor lighting conditions. Pho-tos may be improved using image enhancement tools, butmanually retouching every single photograph is infeasible.Auto-enhancement tools are available (e.g., from Picasa orWindows Live Photo Gallery), but as Kang et al. [8] haveshown, users can have significantly different preferencesthat are not reflected in such tools.

We use Kang et al.’s observations to explore a new ap-proach for image enhancement. We explore if groups ofusers have similar preferences. Our hypothesis is that suchclusters do exist and that we can utilize that informationin order to collaboratively enhance images. Intuitively, weexpect that enhancement preferences should be similar forusers with similar taste on similar images and consequentlywe can derive methods that would let us harness the en-

hancement efforts of multiple users to build predictive mod-els. Such adaptation of the image enhancement problemto the collaborative setting is further strengthened by theever growing use of web-based photo-sharing systems suchas Facebook and Flickr. Adding a personalized image en-hancement component to those sites would allow the systemto suggest appropriate enhancement parameters for new im-ages by utilizing prior enhancements made by other userswith similar preferences.

Our work has three key contributions. First, we describea comprehensive user study through Amazon MechanicalTurk in order to collect image enhancements from a largenumber of users. Second, we propose a probabilistic modelthat explicitly encodes clusterings of user preferences andderive an efficient inference method to collaboratively en-hance unseen images for novel users. Finally, we empir-ically show that explicitly modeling the clustering of userenhancement preferences leads to better predictions of en-hancement parameters than the existing one touch-buttoncommercial auto-enhance functionalities.

2. Previous WorkThe basic idea of image-derived priors is to exploit in-

formation in a predefined image set to automatically deter-mine the most likely parameters for enhancing an image. Ithas been used for a wide range of tasks including denoising[3], color correction [13], and image restoration [2]. Theseapproaches use generic data sets with a large number of im-ages that do not provide any information about user identityor user preferences. Grabler et al. [4] use sample imagemanipulations to learn macros to be applied to images withsimilar content. Also, Joshi et al. [7] narrowed the domainof prior images to a person’s favorite photographs to de-velop a personal image enhancement framework. In princi-ple, these methods are user-dependent, but no studies weredone to establish if their variance across users is significant.

The work closest to ours is that of Kang et al. [8], whoproposed a model for personalization of image enhance-ment that learns user preferences by observing her choicesin a training set. They conducted a user study that showed

249

users

images

Building users database Discovering clusters Enhancing new images

User in

Cluster 1?

User in

Cluster 2?

m

n

m

n

m

n

u

u

xxx

2

11

11

21

?

?

}{ ,, ki uu }{ ,, pj uu

cluster 1 cluster 2

1

1y

1

ny

2

1y

2

ny

Enhancement

Parameter

Enhancement

Parameter

Image Image

Figure 1. Overview of collaborative personalization of image enhancement. A database of users is built, with θui the observed enhancement

vector for user u and image xi. Enhancements are not observed for every combination of images and users, which is indicated by thesymbol ‘?’ in the matrix. An algorithm is used to discover cluster membership for users, and to infer enhancement vectors yc

i associatedwith cluster c for each ith image. New images are enhanced based on similarities to existing images and cluster to which the user belongs.

that different users do have different preferences in imageenhancement, and demonstrated that personalization is animportant component to improve the subjective quality ofimages. Our method builds upon both personalization ofimage enhancement and collaborative filtering techniques.For personalization of image enhancement, our approachasks users to enhance a set of training images using an in-tuitive interface to collect preference information. In thisportion of our work, we implemented methods similar tothose in Kang et al. [8]. However, our work substantiallyextends that personalization approach to consider the con-tribution of other users with similar preferences.

Collaborative filtering is an approach to build recom-mender systems, which analyzes patterns of user inter-est in products to provide personalized recommendations[9]. Intuitively, collaborative filtering works by building adatabase of preferences for items by users, and matchingnew users against this database to find neighbors with sim-ilar preferences [11]. Most existing collaborative filteringmethods attempt to predict a scalar quantity, and thus arehard to adapt to our scenario where our goal is to predicta vector of enhancement parameters. This is the first workthat we know of that proposes the idea of using the wisdomof crowds to enhance images via collaborative filtering.

3. OverviewFigure 1 summarizes the flow of tasks towards building

a system for collaborative image enhancement. We first ac-quire a database of images that are enhanced by multipleusers, and represent the database as a table (leftmost of Fig-ure 1) whose rows are individual users and columns are im-ages. Each entry in the table corresponds to enhancementparameters associated with a different user u and image xi,and are represented by vector θui . The images used for train-ing are a fixed set, and are reasonably representative of im-ages that need enhancement. Note that this set of represen-tative images is large; hence, it is unreasonable to expect

that each user will provide enhancement parameters for allthe images. Instead, every user enhances only a small sub-set of images, and as such, there are quite a few entries inthe table that remain unobserved (denoted as ’?’).

Once this database is acquired, we analyze it to learn astatistical model that explicitly encodes clustering of users.Specifically, we recover groupings of users and estimate thepreference parameters of each individual group for all therepresentative images (middle of Figure 1). Once such astatistical model is learned, we can enhance a new (i.e., un-seen) image for any cluster by considering its similarity tothe images in the representative set.

4. Building BlocksPrior to running our user study, we need to pick the knobs

required to perform image enhancement and select our rep-resentative set of training images. Also in order to effec-tively perform collaborative filtering, we require a distancemetric between images that reflects differences in the re-spective enhancement operations required on those images.Enhancement Operations: We considered two global en-hancement operations: contrast manipulation (S-curve andGamma correction) and color correction (tint and temper-ature), which as with Kang et al. [8], are represented by 5enhancement parameters (two parameters for S-curve, oneeach for gamma-correction, tint and temperature).Learning a Distance Metric between Images: One of thekey ingredients in our approach is distance (or similarity)between images. In particular, given two images, we wantto estimate the extent to which these images require sim-ilar enhancement parameters. We approach this as a met-ric learning problem, in which a distance between visualfeatures is adapted to include information of enhancementparameters. For every image in the photo collection, weextract 6 color histograms from the RGB channels and theHSV components, which are used to build a unique imagedescriptor. Formally, let g(xi) and g(xj) be visual feature

250

vectors for two images xi and xj . We use the Mahalanobisdistance in the visual feature space:

dA(xi,xj) = (g(xi)− g(xj))TA(g(xi)− g(xj)) (1)

which is parameterized by a square matrix A to encodeweights associated with different features and correlationsamongst them in order to reflect the disparity in the en-hancement parameter space. We learn the matrix A usingan online algorithm following the methodology proposedby Jain et al. [6]. The target distance between any two im-ages is considered to be the distance between their auto-enhancement parameters. Specifically, we generate con-straints of the form (g(xi), g(xj), δij), where δ is the dis-tance between the enhancement parameters (obtained froman auto-enhance utility) of the corresponding images. Thisspecific online algorithm is well suited for our task, becauseour image collection has approximately 100 million con-straints, which makes it non-trivial for other distance metriclearning methods to handle.Representative Image Selection: Selecting a good setof representative images is critical. We downloaded a setof real photographs from Flickr, following a “samplingthrough time” strategy, in which the parameter “date taken”is randomly changed [1, 5]. The list of keywords includes“landscape,” “nature,” “people,” and “sports,” among oth-ers, to cover a diverse range of photographs without focus-ing on particular objects. We filtered out images that are toosmall (images smaller than 800× 800 are removed), black-and-white, and have very little intensity variation (measuredby histogram entropy). In the end, our downloaded set con-sisted of more than 300,000 photos.

Given the set of images and the distance metric betweenimages, we need to identify a subset of different images thatrepresent the original set. This problem was approached asa sensor placement problem, in which a budget of sensorsis available to be located in a space, deciding which posi-tions to chose to maximally cover spatial monitoring. In ourproblem, the sensors are the available images in the givenset where the similarity (covariance function) is defined bythe kernel Kij = exp(−dA(xi,xj)/mean(dA(:))). Weused the algorithm described by Krause et al. [10], in whichthe elements that maximally reduce the entropy in the orig-inal set are iteratively selected.

In the Flickr snapshot, we also observed that users tendto upload many similar images. We applied the sensorplacement algorithm to images from the same user, to keeponly the three most “different” images from each individ-ual. We then combined these filtered images and ran the al-gorithm again to select a final photo collection for the userstudy. We found that with 200 images, we kept about 84%of the information from a set of approximately 25,000 can-didates. Figure 2 shows how the information gain reducesas long as informative images are being selected. These

0

0.2

0.4

0.6

0.8

1

1.2

0 500 1000 1500 2000 2500

200 images = 84% information

(from 25,000 candidates)

Number of selected images

Info

rmati

on

Gain

Image Selection via Sensor Placement Algorithm

Figure 2. The image selection algorithm picks images that pro-vide maximum increase in mutual information. This results in adecreasing information gain in the selected image set.

200 images sufficiently represent the visual feature spacedefined by the metric that we have learned.

5. User StudyThe goal of this study was to build the user-image ma-

trix of enhancement parameters. We opted for Amazon Me-chanical Turk (MT) to engage people in our image enhance-ment task. Each person had to enhance a small number ofimages by selecting the most preferred enhanced version viaan interactive web system. We managed to collect enhance-ment parameters from 336 valid users.

Two main criteria guided the design of the system for im-age enhancement used in this study: (1) easy of use and (2)web-oriented. For the first criterion, the user interface doesnot require parameter tweaking to apply enhancements. Weimplemented a user interface similar to the one proposed byKang et al. [8], in which the system presents a 3×3 matrix,with an enhanced version of the image in each cell. In thefirst iteration, the system presents the original image in thecenter cell surrounded by 8 possible initial enhancements.When the user clicks one of these possible enhancements,new similar enhancements are computed to replace the cur-rent ones. When the user has identified the most preferredenhanced version, the system records the enhancement pa-rameters that produce that image and presents another one.Figure 3 shows a screenshot of this user interface.

For the second design criterion, the application is re-quired to be a web-oriented system, to allow people to en-hance images online through MT. We opted to deploy ourapplication using cloud computing services, which allowsour system to scale up as needed to compute image en-hancements online. In our user study, we asked each partic-ipant to enhance 20 images, which are randomly assignedfrom the collection of 200 photos. The assignment of im-ages to users attempts to have approximately the same num-ber of users enhancing each image. The maximum time al-lowed to complete this task was 1 hour and the reward was

251

Figure 3. Web-based user interface for image enhancement.

0

0.2

0.4

0.6

F M

18 - 24

25 - 34

35 - 44

45 - 54

55 - older

Gender

Age

None

Basic

Intermediate

Professional

0%

10%

20%

30%

40%

High

School

Some

college

Bachelour

degree

Masters Ph.D

Never

Year

3 months

Month

Week

Daily

Education level

Expertise in Photography Camera use

Figure 4. Demographics of subjects involved in the MT study.

US$1.50. The click count, click rate, and time stamps wererecorded to identify spammers.

Right before starting the task, participants were asked tofill a questionnaire aimed to collect demographic informa-tion, including gender, age bracket, country, and educationlevel. They also have to indicate their level of expertise inphotography and frequency of camera use. Figure 4 showsthe proportions of these attributes for 336 valid subjects.Most of the subjects are from USA (43%) and India (41%),with the rest from more than 20 other countries.

6. Enhancing Images CollaborativelyLet us denote the collection of all the n images as X =

{xi}. The user study provides us with sets of enhancementparameters chosen by all the m users. Let θui be the en-hancement parameters chosen by user u for image xi. Be-cause of resource and time constraints, every user enhancesonly a small subset of images (20 images out of 200) andconsequently there are a lot of images for which we do notdirectly observe the enhancement parameters correspond-ing to each user. Our goal is to derive a framework thatcan (1) infer these missing enhancement parameters for ev-ery user in the study, and more importantly, (2) determineenhancement parameters for new images and users.

Our model is motivated by the methods in collaborativefiltering. In particular, the first key underlying assumptionhere is that similar users should have similar enhancement

parameters for similar images. Assumptions like this arealso at the heart of many collaborative filtering methodolo-gies; we could in fact adapt those methods to infer missingenhancement parameters for all the users in the study. How-ever, most of the work in collaborative filtering focuses onpredicting a scalar quantity while our goal is to model avector of enhancement preferences. While off-the-shelf ap-proaches can be individually adapted to each enhancementparameter, such simplistic scheme ignores the structure ofrelationships across parameters and will be sub-optimal.

In this work, we extend the collaborative filtering to asetting of structured prediction where we jointly predict allthe components of the enhancement preferences. Specifi-cally, our model not only encodes similarity across imagesand users but also models relationships between differentcomponents of the enhancement space. The notion of userssimilarity in our system corresponds to grouping of usersinto clusters. Thus, users are clustered if they have similarenhancement parameters for all the images. In the next sub-sections, we describe (1) a probabilistic graphical model forjointly predicting enhancement preferences that explicitlyencodes similarity across images and groups users into clus-ters, (2) an efficient inference algorithm, and (3) extensionsto do predictions on unseen images and users that were notpart of the user study.

6.1. Probabilistic Model for Enhancements

We propose a model that encodes the dependence of en-hancement parameters on image content as well as userpreferences. Specifically, given the collection of images andclustering of users, we assume that there are latent enhance-ment preference vectors yci which correspond to cluster cand image xi. Further, the enhancement parameter vectorswe observe in the MT study are simply noisy versions ofthese latent true enhancement preferences. Figure 5 illus-trates the factor graph corresponding to the proposed model.The observed enhancement preferences θui from differentusers are denoted as circles and we introduce a discrete ran-dom variable (squares) hu, u ∈ {1, ..,m} for each user thatindicates the cluster the user belongs to. We also use Yi

to denote collection of all true enhancement preferences forthe ith image across all the clusterings. The shaded nodescorrespond to random variables that are observed. For ex-ample, in Figure 5, the enhancement preferences for user1 are known for images 2 and n, while user m did not en-hance image 2. This is consistent with our data set whereusers enhance only a subset of all training images.

The model also imposes smoothness constraints usinga GP prior [12] in order to account for the image content.We use a Gaussian Process prior to enforce the assump-tion that “similar” images should have similar preferenceparameters. In particular, for each cluster c we induce theGP prior (denoted as GP (c)), where each of the five com-

252

𝑌1 𝑌2 𝑌𝑛

𝜃2 1

𝜃1 1

𝜃𝑛 1

𝜃2 m

𝜃1 m

𝜃𝑛 m

User 1 User m

ℎ1 ℎ𝑚

𝑝(Y|Χ)

𝑝 𝜃𝑖 𝑌𝑖 , ℎ1 = 𝜙ℎ1(𝜃𝑖 , 𝑌𝑖) 1 1

𝑝 𝜃𝑖 𝑌𝑖 , ℎ𝑚 = 𝜙ℎ𝑚(𝜃𝑖 , 𝑌𝑖) m m

Figure 5. Factor graph depicting the proposed model. Shadednodes correspond to observed preferences. Not all observationsare available for all the images and users. Square boxes depictlatent random variables corresponding to cluster membership.

ponents of the latent variables yci and ycj are assumed to bejointly Gaussian with zero mean and the covariance spec-ified using a kernel function applied to xi and xj . For-mally, GP (c) ∼

∏5p=1N (yc(p); 0,K). Here yc(p) is

the column vector of pth component of enhancement pref-erence for all images corresponding to the cluster c andK is a kernel matrix1 with Kij = k(xi,xj) and encodessimilarity between pairs of images. In this paper, we useKij = exp(−dA(xi,xj)/mean(dA(:))). Note, that the GPprior above does not explicitly encode the relationship be-tween the components of the parameters and we can assumethat the components of yci have been whitened (zero mean,with unit covariance) beforehand2. Also, note that all the di-mensions of the latent variable Y are coupled in the modeland performing inference will preserve the relationships be-tween different components.

Let Θ be all the enhancement preferences from all theusers and for all the images. Our proposed model induces aconditional probability distribution p(Θ,Y,h|X) using theGP prior p(Y|X), prior probabilities on the cluster mem-bership p(h), and the potential terms p(θui |xi,Yi, hu) thatlink the latent image preferences to the ones that are ob-served. Thus, the conditional distribution induced by ourmodel can be written as

p(Θ,Y,h|X) =1Zp(Y|X)p(h)p(Θ|X,Y,h)

=1Z

k∏c=1

GP (c)m∏u=1

p(hu)n∏i=1

φhu(θui ,Yi),

1This kernel matrix is a positive semidefinite matrix and is akin to thekernel matrix used in classifiers such as SVMs.

2In this work, we whiten the enhancement preferences before applyingthe model and re-project back to orginal space eventulally which results inpreservation of the structure of the parameter space.

where Z is the partition function (normalization term) andthe potential φhu(θ

ui ,Yi) corresponding to a user u and im-

age xi takes the following form:

φhu(θui ,Yi) ∝ e−

||yhui−θui ||

2

2σ2 . (2)

Here, yhui are the hidden random variable for the samecluster as the cluster indicated by hu and the image xi, andσ2 is the noise parameter that determines how tight the re-lation between the smoothness constraint and the final la-bel is. By changing the value of σ we can emphasize orde-emphasize the effect of the GP prior. In summary, themodel provides a powerful framework for encoding depen-dence of enhancement parameters on image content (via theGP prior) as well as the clustering of users and allows us tocombine the prior assumptions with the data that is observedin the MT study.

6.2. Inference in the Model

Given the observations Θo from the MT study the keytask is to infer the posterior distribution p(Y,h|X,Θo)over latent true enhancement preferences and the clusteringmembership for all the users. Performing exact inference isprohibitive as the joint distribution is a product of a Gaus-sian (GP prior and the φ(·) potentials) and non-Gaussianterms (cluster membership). We resort to approximate in-ference techniques in order to get around this problem. Inparticular, we perform an approximate inference by maxi-mizing the variational lower bound with the assumption thatthe posterior over the unobserved random variable Y and hcan be factorized:

F =∫

Y,h

q(Y)q(h) logp(Y,h|X,Θo)q(Y)q(h)

≤ log∫

Y,h

p(Y,h|X,Θo),

where q(Y) =∏kc=1

∏5p=1 q(y

c(p)) is assumed to be aGaussian distribution and q(h) =

∏mu=1 q(hu) is a discrete

joint distribution over the unobserved labels. The approx-imate inference algorithm aims to compute good approxi-mations q(Y) and q(h) to the real posteriors by iterativelyoptimizing the above described variational bound. Specif-ically, given the approximations qt(Y) and qt(h) from thetth iteration and assuming uniform prior over p(h) the up-date rules are:

qt+1(yc) ∝ GP (c)∏

θui ∈Θo

[φc(θui ,Yi)]qt(hu=c)

qt+1(hu = c) ∝∏

θui ∈Θo

φc(θui ,mean(qt+1(Yi))).

Intuitively, the update of image enhancement preferencesconsiders the cluster membership from the previous itera-tion and uses it to decide if a data term should be included

253

in update for each cluster. Similarly, the update for the pos-terior over the cluster membership considers mean enhance-ment preferences from the previous iteration. Thus, startingfrom a random initialization the parameters and posterior ofthe cluster memberships are iteratively updated untill con-vergence. Upon convergence, we obtain posterior distribu-tion of cluster membership for each user (q(h)) as well thedistribution over the true image enhancement preferences(q(Y)) approximated as a Gaussian distribution.

6.3. Handling New Images and Users

Besides inferring about the images in the database, weare also interested in predicting enhancement preferencesfor an image xtest that was not in the original training setas well as for a user who was not part of the user study.

The more straightforward case is where we infer en-hancement preferences for a user u who was a part of theuser study and had enhanced training images for us. Here,we simply need to run the inference algorithm where weaugment the random variable Y with Ytest, the collectionof random variables that represent true enhancement param-eters for the new test image across all the clusterings. Thus,the new image can be enhanced using the mean of inferredenhancement preferences corresponding to the cluster theuser u belongs to. Note that if we have already performedvariational inference on the training data, inference after theaugmentation only requires one iteration of update equa-tions. This is because the new image does not introduce anynew information about clusterings of the user or enhance-ment parameters, hence, making it fairly efficient.

It is trickier if we need to enhance images for a user whois not part of the study. However, once we know the clustermembership of the new user, we can simply use the schemementioned above to estimate the enhancement preferences.Thus, the problem really reduces down to establishing thecluster membership of the user. To this end, we can employseveral methods. For example, the new user can enhancesome of the images from the training set, which in turn willgive us evidence about membership. The user now can beconsidered a part of the available corpus and inference canbe run as before. Alternatively, we can also engage the userin an introductory dialogue where training images are used;the resulting enhancements will indicate membership. Inthis work, we use the former approach of asking users toenhance a subset of training images to test the extension ofthe system to new users and images.

7. Experiments and ResultsWe first perform experiments to determine the correct

number of clusters using the data collected in the MT study.Train-test splits were created by randomly choosing 10%of the observed enhancement vectors in the user-image ma-trix as test examples. The other 90% is used to run the in-

1.238

1.243

1.248

1 2 3 4 5

RM

SE

Number of Clusters

Error in Parameter Space

8.6

8.8

9

9.2

1 2 3 4 5

RM

SE

Number of Clusters

Error in Intensity Space

Figure 6. Average estimation error on training for different numberof clusters.

ference algorithm described in Section 6.2 with σ2 fixed to10−3. Evaluating the quality of user adjustments is difficult,since there is no universal agreement on the perceptual met-ric. We resorted to the objective measure of intensity differ-ence looking at average Root Mean Square Error (RMSE)in parameter space (enhancement vectors) as well as imageintensity space (resulting enhanced images).

Figure 6 highlights the variation of RMSE as we changethe number of clusters in the model. Note, that consider-ing only one cluster is equivalent to assuming that all usersfollow the same preferences for enhancing images. We ob-serve that error is highest in both parameter and intensityspaces when the number of clusters is fixed to one andreduces significantly as we start incorporating additionalnumber of clusters. This result is consistent with the obser-vations reported by Kang et al. [8], where it was shown thatdifferent users do have significanlty different preferences.We also find that the error was minimum in the parameterspace for 3 clusters and pretty close to minimum in the in-tensity space. Thus, we perform rest of the analysis andexperiments by fixing the number of clusters to 3.

The proposed method can be thought of as a clusteringalgorithm with constraints imposed by similarity in the im-age appearance space. Consequently, it is in principle thesame as an EM algorithm for learning a Gaussian mixturemodel, and it does not prefer that most users have their owndistinct cluster. We think that the increase in error due tofurther addition of clusters might be due to the small num-ber of enhancements operations allowed and we can expectto see different clusterings if additional operations are per-mitted.

Figure 7 shows example images enhanced according tothe preferences of users in each of the three clusters, giv-ing a general indication of the preferred appearance in eachcluster. While the corrected images in cluster 2 are slightlymore saturated than the others, images in cluster 3 seemto have more contrast. A more quantitative description ofthe preferred enhancements in each cluster can be seen inFigure 8, which presents the distribution of the five param-eters as box plots. The box plots are over the means of theinferred distribution over Y (all 200 images for the 3 clus-ters), where the red line in each column denotes the me-dian. Note that the largest difference across clusters is forthe Power Curve and the S-Curve Inflection Point, suggest-

254

Cluster 1 Cluster 2 Cluster 3

Figure 7. Example images enhanced according to the preferencesdiscovered for each cluster.

ing that variations in contrast is a dominating factor acrossthe clusters. There are some differences across the rest ofthe three parameters as well suggesting a difference in pa-rameter choice amongst the users.

In the scatter plot shown in 9, each point correspondsto an enhancement preference color-coded by the cluster towhich it was assigned. Note that the points are clusteredfairly evenly in this 2D projection of the space, further em-phasizing the latent structure which is successfully discov-ered by the inference algorithm. We also analyzed statis-tics of demographic data across the clusters but the statisticswere fairly similar across the clusters.

Next, we evaluate the ability of the proposed model topredict enhancement preferences for a new user on unseenimages in a collaborative environment. In this test, 10% ofall users were randomly chosen as test subjects and infer-ence was performed using only the data observed for restof the users. We simulate the real-life situation where everytest user provides evidence about its cluster membership byenhancing some images. Since this work is about collabo-ratively using information about image enhancement avail-able from many different users, we show results on scenar-ios where a test user has provided such little informationthat personalized model of [8] cannot learn. We look at theRMS error in intensity space as each user enhances one im-age at a time (chosen randomly) and compare them with re-sults of auto-enhancement from Picassa and Windows LivePhotogallery.

Figure 10 presents the plot of RMSE obtained by the

Figure 9. Scatter plot of all images according to parameters Power-Curve and S-Curve Inflection Point (which partially control globalimage contrast). Cluster 1 in red squares, Cluster 2 in green circlesand Cluster 3 in blue triangles.

8.5

8.7

8.9

9.1

9.3

9.5

9.7

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

RM

SE

(In

ten

sity

) on

Tes

t S

et

Number of Training Images Enhanced by the Test User

Comparison with Existing Systems

Our System (3 Clusters)

Picasa

Photogallery

Figure 10. Average error of predicting personalized enhancementparameters. Auto-enhancement tools do not learn user prefer-ences. Our approach predicts personalized enhancements with in-creasing accuracy as users provide new examples.

system with increasing number of enhancements availablefrom the user. We find a consistent reduction in estimationerrors as more enhancements are added to the personal pro-file. This is due to the fact that additional enhanced imagesprovides more information about the cluster membership ofthe user, enabling the inference procedure to estimate a bet-ter recommendation. We also observe that the performanceof the collaborative approach is far better than the two auto-enhancement tools. While all the three approaches performsimilarly when there is no evidence about the cluster mem-bership of the user, we see that the performance of collab-orative strategy greatly improves as user starts enhancingimages. Also, note that the gains start showing up with asfew as 2 images indicating that strong gains can be obtainedwith the collaborative enhancement of images.

Finally, Figure 11 shows several image examples withthe corresponding enhanced versions produced by Photo-

255

Power S Curve Lambda S Curve Inflection Temperature Tint

Figure 8. Box plots of values for each parameter (first three control global contrast, last two control color correction). The x-axis indicatescluster number and y-axis corresponds to range of values. Red lines denote the median, end lines are at the lower and upper quartile values,and crosses beyond the ends of the whiskers are outliers.

Original Image PhotoGallery Picassa Cluster 3

Figure 11. Example images with the corresponding enhanced ver-sions generated by auto-enhancement tools and the proposed ap-proach for a subject that belongs to cluster 3.

Gallery and Picassa. Also, the enhanced version of ourmethod, for a subject of the cluster 3 is presented. In gen-eral, each tool produces a different enhancement resultingin a different final look and feel. Notice that the suggestionmade by our method has been inferred using the subjectivepreferences learned from people in cluster 3, thus it is morelikely to be preferred by a user belonging to that population.

8. Conclusion and Future Work

We present a novel approach for image enhancement thatfollows a collaborative framework to recommend correc-tions based on user preferences. Instead of building a sys-tem that is trained to enhance images for a specific user, ourtechnique is a principled and practical way for collabora-tion at a web scale, and can encompass alternative param-eterizations and error criterion. The main idea is we pre-discover the clusters of personalized enhancement param-eters through collaborative learning on large image collec-tions. Then, for a new user, we need only figure out whichcluster he/she belongs to.

Results of the cluster analysis showed the existence ofthree main groups, mainly characterized by differences incontrast preferences. Experimental results indicate that thecollaborative enhancement strategy significantly helps inmaking better predictions of enhancement parameters thanexisting one touch-button commercial auto-enhance tools.

We do not claim that our results extend to a much largerscale with many more users and a much richer set of en-hancement knobs; more experiments are required to sub-stantiate such a claim. Future work includes active selec-tion of images that would enable better estimation of clustermembership with fewest image enhancements by the user.

References[1] T. S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng.

Nus-wide: a real-world web image database from nationaluniversity of singapore. In CIVR ’09. ACM, 2009.

[2] K. Dale, M. K. Johnson, K. Sunkavalli, W. Matusik, andH. Pfister. Image restoration using online photo collections.In ICCV. IEEE, 2009.

[3] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T.Freeman. Removing camera shake from a single photograph.ACM Trans. Graph., 25(3):787–794, 2006.

[4] F. Grabler, M. Agrawala, W. Li, M. Dontcheva, andT. Igarashi. Generating photo manipulation tutorials bydemonstration. ACM Trans. Graph., 28(3):1–9, 2009.

[5] J. Hays and A. A. Efros. Scene completion using millions ofphotographs. Commun. ACM, 51(10):87–94, 2008.

[6] P. Jain, B. Kulis, I. Dhillon, and K. Grauman. Online metriclearning and fast similarity search. In NIPS, 2008.

[7] N. Joshi, W. Matusik, E. H. Adelson, and D. J. Kriegman.Personal photo enhancement using example images. ACMTrans. Graph., 29(2):1–15, 2010.

[8] S. B. Kang, A. Kapoor, and D. Lischinski. Personalizationof image enhancement. Computer Vision and Pattern Recog-nition Conference, CVPR, 2010.

[9] Y. Koren, R. Bell, and C. Volinsky. Matrix factorization tech-niques for recommender systems. IEEE Computer Journal,pages 30–37, 2009.

[10] A. Krause, A. Singh, and C. Guestrin. Near-optimal sen-sor placements in gaussian processes: Theory, efficient algo-rithms and empirical studies. J. Mach. Learn. Res., 9:235–284, 2008.

[11] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Item-basedcollaborative filtering recommendation algorithms. ACM,WWW, 2001.

[12] M. Seeger. Gaussian Processes for machine learning. Inter-national Journal of Neural Systems, 14(2), 2004.

[13] R. Stanikunas. Investigation of color constancy with a neuralnetwork. Neural Networks, 17(3):327–337, April 2004.

256