Separation of Highlight Reflections on Textured Surfaces

Ping Tan1∗ Stephen Lin2 Long Quan1

1Hong Kong University of Science and Technology2Microsoft Research Asia

Abstract

We present a method for separating highlight reflectionson textured surfaces. In contrast to previous techniques thatuse diffuse color information from outside the highlight areato constrain the solution, the proposed method further cap-italizes on the spatial distributions of colors to resolve am-biguities in separation that often arise in real images. Forhighlight pixels in which a clear-cut separation cannot bedetermined from color space analysis, we evaluate possibleseparation solutions based on their consistency with diffusetexture characteristics outside the highlight. With consid-eration of color distributions in both the color space andthe image space, appreciably enhanced separation perfor-mance can be attained in challenging cases.

1 Introduction

In various applications such as stereo reconstruction, vi-sual recognition, and tracking, an object is commonly as-sumed to have a consistent surface appearance in differentimages. The appearance of a surface, however, can signifi-cantly vary due to the presence of highlights, which obscureunderlying surface details and appear as additional featuresthat are not intrinsic to the object. Since these sharp reflec-tions of light can be misleading to computer vision algo-rithms, it is generally beneficial to remove them from im-ages.

Previous methods for highlight removal have primarilybeen based on the dichromatic reflection model [11], whichdescribes the color of highlights as a linear combination ofthe specular illumination color and the underlying diffusecolor of the surface. To separate specular from diffuse re-flections, various approaches have been presented for gain-ing information about the diffuse and specular colors, whichthen allows their individual contributions to the observedhighlight color to be determined.

∗This work was done while Ping Tan was a visiting student at MicrosoftResearch Asia.

Several techniques extract data for highlight removalfrom multiple images captured under changing viewingconditions, such as different polarizer angles [9], lightingdirections [10], and viewpoints [7]. These methods havebeen effective in separating diffuse and specular reflections,but often in practice such image sets are not available.

Acquiring separation information in the more generalcase of a single input image presents a challenging problem.Prior single-image methods have inferred diffuse colors intwo ways. One is by examining the expected diffuse colorsof neighboring pixels in the image. Tan and Ikeuchi [14]remove highlights by iteratively shifting chromaticity val-ues towards those of greater maximum chromaticity amongneighboring pixels. Tan et al. [12] also present an iterativetechnique that uses diffuse chromaticity values of neighbor-ing pixels to guide highlight removal. In these neighbor-based methods, diffuse information is largely being propa-gated from outside the highlight towards the inside. Thistype of approach may encounter problems from disconti-nuities in surface colors, across which diffuse informationcannot accurately be transferred.

The other approach to single-image highlight removal isto analyze the distributions of image colors within a colorspace. Klinker et al. [3] identify linear clusters of dif-fuse colors and specular colors in the RGB space. Avoid-ing color segmentation, Tan and Ikeuchi [13] project im-age colors along the illumination color direction to a pointof lowest observed intensity to determine the diffuse color.Such approaches based on color distributions can be greatlyimpaired by clutter in the color space that is caused by anumber of factors, including image noise, color blendingat edges, and numerous image colors. This clutter, exempli-fied in Fig. 1, makes it difficult to distinguish various diffuseand specular colors. Furthermore, as noted in [13], pixelswith distinct diffuse colors may be projected along a givenillumination color direction to the same diffuse color. Thisambiguity between color variations caused by specular re-flections and those of diffuse texture is a general problem ofcolor space analysis.

In this work, we present a separation method that takesgreater advantage of image space information for color

(a) (b) (c)

Figure 1. Color space clutter for the object in Fig. 4. (a) Color space of original image. To solve for the diffuse color D of apixel, its highlight color I (blue dot) may be projected along the illumination color direction S (yellow line) onto a point having thesame chromaticity as a diffuse color in the image. (b) Color space of ground truth diffuse image, captured using cross polarization.(c) Examination of color space points along the projection line from diffuse areas of the original image. Due to clutter, the correctprojection point becomes ambiguous. The ground truth value for this pixel is the green dot.

space analysis. While previous techniques are based uponRGB values from outside a highlight, the proposed methodadditionally examines higher-order color data in the form ofspatial color distributions. This color texture data from out-side the highlight can provide valuable information in de-termining diffuse surface colors within the highlight. Withthis information, we formulate constraints on the separationproblem that can effectively resolve ambiguities that existin color space methods. Unlike in neighbor-based separa-tion techniques, this use of spatial information in the imagedoes not suffer from surface color discontinuity problems.

In employing texture-based constraints, an issue thatmay arise is texture variation, where some local diffuse tex-ture within a highlight area may not be fully consistent withoutside textures. In contrast to reconstructing the diffusecolors of a highlight region from diffuse textures in the im-age, our method utilizes this spatial color information onlyas an additional constraint to resolve separation ambigui-ties in an individual pixel. When the underlying diffusetexture in a highlight has greater consistency with the ob-servable diffuse texture in an image, the proposed techniquecan better exploit the texture information to reduce ambigui-ties. If no clarifying texture data is identified for a pixel, ourmethod computes its separation in the same manner as tra-ditional color-space techniques. Our experiments show thateven for highly complicated textures, the consideration ofspatial color information can lead to appreciable improve-ments in separation.

2 Highlight Formation

Before presenting the details of our method, we give abrief review of highlight formation and describe how colorspace analysis can be used to separate highlight reflections.

Reflections from a surface can be classified as beingeither diffuse or specular. Diffuse reflection traditionallyrefers to light that has entered a material volume, scattered

among particles in the medium, and then exits the volume.From interaction with the material, the color of the lightbecomes influenced by the intrinsic color R of the surface.In contrast, specular reflection describes light that reboundsdirectly off the surface without entering the volume, and ex-hibits the color E of the illumination source [4].

The dichromatic reflection model [11] describes re-flected light color I as a linear combination of diffuse andspecular colors:

Ic = ρd

∫

λ

E(λ)R(λ)qc(λ)dλ + ρs

∫

λ

E(λ)qc(λ)dλ, (1)

where qc is the spectral sensitivity of color filter c ∈{R, G, B}, ρd and ρs represent coefficients that govern themagnitude of diffuse and specular reflection, and λ denoteswavelengths of light in the visible spectrum.

Equation 1 can be expressed more simply as

Ic = ρdDc + ρsSc (2)

where D and S respectively represent the diffuse and spec-ular colors. In highlight removal, the values of these modelparameters are estimated for each pixel, so that the specularcomponents ρsS can be subtracted from the image.

If D and S can be determined, then the values of ρd andρs can be computed from the dichromatic reflection model,which provides three equations (c = R, G, B) for the twounknowns. Supposing that the specular illumination colorhas been obtained, such as from color constancy [2, 15], theseparation problem becomes one of deducing D for eachpixel.

From color space analysis such as in [13], the diffusecolor D of a pixel can be inferred by projecting its highlightcolor I along the specular color direction S onto a pointthat has the same chromaticity (normalized color) as a dif-fuse color present in the image, as illustrated in Fig. 1. Aunique solution for D is obtained if only one such diffuse

p

p’1D′

nnD ×′1I

nnI ×

S

1I

1D′S

nnI ×

nnD ×′

(a)

S

I

S

I

S

I

(b)

1x1 3x3 5x5

Figure 2. Constraints based on diffuse texture information. (a) Diffuse color estimation with texture windows. (b) Resolution ofambiguities at increasing texture scales. Above: window areas within highlight. Below: distribution of color space points that lieclose to the illumination constraint line.

color exists along this projection line. However, in prac-tice a number of diffuse colors may be present due to colorspace clutter, and this leads to ambiguity in the separationsolution.

3 Texture-based Constraints

To resolve this ambiguity, our method utilizes color tex-ture information from outside the highlight area. Texturesdescribe spatial patterns of colors or geometric features thatcharacterize the appearance of surfaces. These distinctivepatterns over a surface provide an important cue for rec-ognizing objects and segmenting image regions, and therehence has been a substantial amount of work on textureanalysis [16]. Our method for highlight separation alsotakes advantage of these characteristic color variations ona surface. As diffuse colors from outside a highlight areahave been used as representatives of diffuse colors on theinside, diffuse color textures can serve as exemplars of theirspatial distributions.

Numerous models have been proposed for texture rep-resentation, such as Markov random fields [17], filter re-sponses [8], and textons [5]. Our highlight separation tech-nique could potentially be adapted to use any of these tex-ture representations, but in this work, we directly utilize thecolor image areas around the highlight.

In explaining our use of texture information, we begin bydescribing how matches are determined between an imagewindow within the highlight and a corresponding windowoutside the highlight area with respect to the dichromaticreflection model. For a 1×1 window, our method is equiva-lent to the traditional color analysis [13] described in Fig. 1.A highlight pixel p with an image color I is matched toa diffuse pixel p′ with a diffuse color D′ if the minimumangle with respect to the color space origin between D′ andthe illumination constraint line defined by I and−→S is small.For this match, the point I − ρsS on the illumination linewhich gives the smallest angle to D′ determines the esti-mated highlight intensity ρs of pixel p. Since the angular

distance between a candidate point p′ and the illuminationconstraint line of p is independent of specular intensity, thismatch is specularity-invariant. Furthermore, it is shading-invariant because angular distance is independent of diffuseintensity.

For an n × n window centered on a highlight pixel p,we determine the matched diffuse pixel for p by comparingeach element Ik, k = 1..n2, in this window to correspond-ing elements in windows centered on pixels p′ outside thehighlight, as illustrated in Fig. 2(a). The texture distancebetween the windows of p and p′ is obtained by averagingthe distances between their corresponding elements:

E(p, p′) =1n2

n2∑

k=1

dist[Dp′(k), Ip(k),−→S

](3)

where k indexes pixels within the window, and dist rep-resents the minimum angular distance between Dp′(k) andthe line defined by Ip(k) and −→S . After the match is de-termined, the approach used for 1 × 1 windows is used tocompute ρs for pixel p.

In real images, there often exist multiple pixels p′ thathave similarly low values of dist at a 1× 1 scale. When thedifferences in Dp′ among such pixels are not small, theircandidate separation solutions for p may differ significantly.To reduce this ambiguity, we utilize the higher order colorinformation in texture. When there is a good match betweenp and p′ over larger local neighborhoods instead of just asingle pixel, the evidence that p′ is a good exemplar for pbecomes stronger, because it is supported by surroundingimage structure that characterizes the surface. On the otherhand, candidate separations for p that are not consistent withtexture at larger scales can be pruned from the set of candi-date solutions.

The benefit of texture analysis at larger scales is exem-plified in Fig. 2(b). In this example, a 1× 1 texture windowfor a highlight pixel yields significant uncertainty in the dif-fuse color. At a 3×3 scale, the higher-order color constraintleads to a reduction in the number of matching pixels. When

(a) (b) (c) (d) (e)

Figure 3. Effects of increasing texture scale, shown for a synthetic image. (a) Original image. (b) Ground truth diffuse image. (c)After 1x1 texture analysis. (d) After 5x5 texture analysis. (e) After 9x9 texture analysis.

the scale is increased to 5 × 5, only a single cluster of dif-fuse color candidates remains, and a separation result canthen be determined with adequate certainty. In Fig. 3, animage sequence illustrates refinements in separation aftereach scale of processing.

While diffuse color matches for highlight pixels gener-ally exist in the traditional color-space case of 1 × 1 win-dows, at increasingly larger texture scales there will likelyexist fewer pixels with matches because of texture varia-tions as well as scale, affine, and projective distortions. Thedescribed approach takes advantage of whatever texture in-formation is usable to deal with separation ambiguities, andexamines windows at multiple scales to promote matching.When no useful texture information is available, our methodyields the basic color-analysis solution.

It should also be noted that the recovered diffuse compo-nent for pixel p is restricted to lie on the illumination con-straint line as in [12]. In contrast to non-parametric methodsfor texture synthesis which also examine patches of texturesamples (e.g., [1, 6]), the estimated diffuse color of p is notdirectly copied from p′, which allows for differences in dif-fuse texture and shading to be maintained. Matches thatare physically possible according the dichromatic reflectionmodel are used only as a reference for disambiguating thediffuse color of a given pixel.

4 Highlight Separation Algorithm

These texture-based constraints for dealing with diffusecolor ambiguities are employed within our proposed algo-rithm for removing highlight reflections. For images withcolor space clutter, previous techniques such as color clustersegmentation [3] or specular-free images [14] for identify-ing highlight locations in a single image may be unreliable.Our method therefore employs user specification of high-light and diffuse regions.

For each pixel, beginning from a texture scale of 1 × 1,a set of candidate diffuse colors along the illumination con-straint line is obtained. Within this candidate set, we deter-mine whether there exists an appreciable ambiguity among

the diffuse colors. If there is ambiguity, the set of candi-dates is iteratively pruned according to information at in-crementally larger texture scales. This process is repeateduntil the ambiguity is resolved. The details of these stepsare described as follows.

Match Determination For a given pixel, windows at thegiven texture scale are examined in the diffuse image areaas described in Sec. 3. To determine whether a diffuse areais a match for the pixel, we compute its distance accordingto Eq. 3. If the distance lies below a specified value, a matchis found, and from its corresponding separation solution weobtain a diffuse color candidate.

Ambiguity Resolution We consider a set of diffuse colorcandidates to be ambiguous if their values lack consistency.This ambiguity is measured as the maximum chromaticitydistance among pairs of diffuse colors within the candidateset. If this distance lies below a specified ambiguity thresh-old, the set is deemed to be consistent. From this set, thecandidate with the smallest angular distance with respect toEq. 3 is taken as the solution.

Otherwise, the match determination step is repeated atthe next higher texture scale with the aim of resolving theambiguity. With the updated set, the ambiguity resolutionstep is repeated by evaluating its consistency.

In some instances of ambiguity resolution, no valid so-lutions remain after an increase in the texture scale. In suchcases, the diffuse color candidates at the previous texturescale cannot be pruned to a compact set of diffuse colorsby texture analysis. For this diverse set of colors, a solu-tion is then determined as the one with the smallest angulardistance.

5 Results

The performance of the proposed algorithm was testedon images captured with a consumer-grade point-and-shootcamera that was radiometrically calibrated. Ground truthseparation results were obtained by cross polarization.

(a) (b) (c)

(d) (e) (f) (g)

Figure 4. Separation results for a painted wooden object. (a) Original image. (b) Ground truth diffuse image, captured using crosspolarization. (c) Ground truth specular image. (d) Diffuse image after processing 1x1 texture windows. (e) Closeup of diffuse areaafter 1x1 processing. (f) Final diffuse image. (g) Closeup of final diffuse area.

Fig. 4(a) displays an image of a painted wooden catwhose color space distribution is shown in Fig. 1(a). Dueto the large amount of color space clutter, traditional colorspace analysis (approximated by the 1 × 1 texture windowprocessing in our algorithm) does not generate accurate sep-aration results, as seen in (d) and (e). After iterations tolarger texture scales, ambiguities are resolved and a higherquality result in (f) and (g) is obtained.

An example with spatially varying specular reflectanceis given in Fig. 5. Surfaces with numerous discontinuitiesin surface color and sharp changes in specular reflection arechallenging for neighbor-based methods, as exhibited in (d).With only 1 × 1 texture window processing, the estimatedseparation is also inadequate as seen in (e). Improvementsdisplayed in (f) are achieved with iterations to greater patchsizes.

In Fig. 6, we display results for an object with smoothlyvarying texture, which presents a difficult separation prob-lem due to significant cluttering of the color space. In addi-tion, the differences in diffuse colors extend approximatelyalong the illumination color direction. Although substantialimprovement from the 1 × 1 result is obtained, some noisein the separation arises due to a lack of texture data outsidethe highlight regions, which cover most of the object.

6 Conclusion

In this work, we presented a highlight separation methodthat takes advantage of available diffuse texture informationto resolve ambiguities in color-space separation solutions.To promote texture matching in the presence of texture vari-ations, our method exhaustively examines textures at mul-

tiple scales throughout the diffuse image region. In someinstances, the texture matches may be sparse due to texturevariation, and consequently the obtained information maybe insufficient to completely resolve ambiguity. However,the consideration of spatial color distributions brings evi-dent improvements in separation results beyond traditionalcolor space analysis.

In future work, we plan to investigate the use of statisti-cal texture models, which may provide a more general rep-resentation of the observed diffuse textures in the image.With a more comprehensive texture model, a more completeand accurate resolution of ambiguities could potentially beobtained.

Acknowledgement

The work is supported by Hong Kong RGC GrantHKUST6182/04E, HKUST6190/05E and Hong Kong UGCgrant AoE/E-01/99.

References

[1] A. A. Efros and W. T. Freeman. Image quilting for texturesynthesis and transfer. In ACM SIGGRAPH ’01, pages 341–346, 2001.

[2] G. D. Finlayson, S. D. Hordley, and P. M. Hubel. Colorby correlation: A simple, unifying framework for color con-stancy. IEEE Trans. Pattern Analysis and Machine Intell.,23, 2001.

[3] G. J. Klinker, S. A. Shafer, and T. Kanade. The measure-ment of highlights in color images. International Journal ofComputer Vision, 2:7–32, 1990.

(a) (b) (c)

(d) (e) (f)

Figure 5. Separation results for a surface with spatially varying specular reflectance. (a) Original image. (b) Ground truth diffuseimage, captured using cross polarization. (c) Ground truth specular image. (d) Diffuse image using the inpainting technique in [12].(e) Closeup of diffuse image after processing 1x1 texture windows. (f) Closeup of final diffuse image.

(a) (b) (c) (d)

Figure 6. Separation results for a surface with smoothly varying texture. (a) Original image. (b) Ground truth diffuse image,captured using cross polarization. (c) Diffuse image after 1x1 processing in our proposed method. (d) Final diffuse image.

[4] H.-C. Lee, E. J. Breneman, and C. P. Schulte. Modeling lightreflection for computer color vision. IEEE Trans. PatternAnalysis and Machine Intell., 12:402–409, 1990.

[5] T. Leung and J. Malik. Representing and recognizing thevisual appearance of materials using three-dimensional tex-tons. International Journal of Computer Vision, 43:29 – 44,2001.

[6] L. Liang, C. Liu, Y.-Q. Xu, B. Guo, and H.-Y. Shum. Real-time texture synthesis by patch-based sampling. ACM Trans.Graphics, 20(3):127–150, 2001.

[7] S. Lin, Y. Li, S. Kang, X. Tong, and H.-Y. Shum. Diffuse-specular separation and depth recovery from image se-quences. In European Conf. on Computer Vision, pages 210–224, 2002.

[8] J. Malik and P. Perona. Preattentive texture discriminationwith early vision mechanisms. Journal of the Optical Societyof America A, 7, 1990.

[9] S. K. Nayar, X. S. Fang, and T. Boult. Separation of reflec-tion components using color and polarization. InternationalJournal of Computer Vision, 21:163–186, 1997.

[10] Y. Sato and K. Ikeuchi. Temporal-color space analysis ofreflection. Journal of the Optical Society of America A, 11,

1994.[11] S. Shafer. Using color to separate reflection components.

Color Research and Applications, 10:210–218, 1985.[12] P. Tan, S. Lin, L. Quan, and H.-Y. Shum. Highlight removal

by illumination-constrained inpainting. In Int. Conf. on Com-puter Vision, pages 164–169, 2003.

[13] R. T. Tan and K. Ikeuchi. Reflection components decompo-sition of textured surfaces using linear basis functions. InComputer Vision and Pattern Recog., pages 125–131, 2005.

[14] R. T. Tan and K. Ikeuchi. Separating reflection componentsof textured surfaces using a single image. IEEE Trans. Pat-tern Analysis and Machine Intell., 27:178–193, 2005.

[15] R. T. Tan, K. Nishino, and K. Ikeuchi. Color constancythrough inverse intensity chromaticity space. Journal of theOptical Society of America A, 21, 2004.

[16] M. Tuceryan and A. K. Jain. Texture analysis. Handbook ofPattern Recognition and Computer Vision, 1993.

[17] S. Zhu, Y. Wu, and D. Mumford. Filters, random fields andmaximum entropy (frame): Towards a unified theory for tex-ture modeling. International Journal of Computer Vision,27:107–126, 1998.