Removing Shadows From Images using Retinex · 2006-03-29 · AppearedInProc.10thColourImagingConference,Scottsdale,AZ2002 1 Removing Shadows From Images using Retinex G. D. Finlayson,

AppearedIn Proc.10thColourImagingConference,Scottsdale,AZ 2002 1

Removing Shadows From Images usingRetinex

G. D. Finlayson, S. D. Hordley M.S. DrewSchool of Information Systems School of Computing Science

University of East Anglia Simon Fraser UniversityNorwich NR4 7TJ Vancouver V5A 1SUnited Kingdom Canada

graham, [email protected] [email protected]

Abstract

TheRetinex Theoryfirst introducedby Edwin Landfortyyearsagohasbeenwidely usedfor arangeof applications.It wasfirst introducedasa modelof our own visual pro-cessingbut hassincebeenusedto performa rangeof im-ageprocessingtasksincluding illuminant correction,dy-namicrangecompression,andgamutmapping.In thispa-perweshow how thetheorycanbeextendedtoperformyetanotherimageprocessingtask: thatof removing shadowsfrom images. Our methodis foundedon a simplemod-ification to the original, path basedretinex computationsuchthatwe incorporateinformationaboutthelocationofshadow edgesin animage.We demonstratethatwhenthelocationof shadow edgesis known the algorithmis ableto removeshadowseffectively. We alsosetforth a methodfor the automaticlocationof shadow edgeswhich makesuseof a 1-d illumination invariantimageproposedin pre-viouswork [1]. In this casethe locationof shadow edgesis imperfectbut we show thatevenso,thealgorithmdoesa goodjob of removing theshadows.

1. Introduction

Edwin Land [2] first proposedhis Retinex Theory ofColourVision forty yearsagoasa modelof how the hu-manvisualsystemprocessesa spatiallyvarying light sig-nal (a retinal image) to arrive at a stableperceptionofcolour at eachpoint in a scene. Sincethat original pa-per themodelhasundergonea seriesof majorandminorrevisionsbothby Land[3, 4, 5] andothers[6, 7, 8, 9] andhasbeenappliedto a rangeof tasks. Its effectivenessasa model for our own visual systemhascontinuedto beexplored[4, 5, 6, 9] whilst at the sametime othershaveconsideredhow thetheorycanhelpin arangeof imagere-productionandimageenhancementtaskssuchasdynamicrangecompression[8, 10], colourcorrection[8, 9, 10] andgamutmapping[11].

In thispaperweexplorehow theretinex theorycanhelpto solve yet anotherimageenhancementtask: that of re-moving shadows from images.Thework we presenthereis closein spirit to thework of others[8, 10] whohavecon-sideredthe imageenhancementcapabilitiesof the theoryandin particularis relatedto the useof retinex for com-pressingthedynamicrangeof animage.In dynamicrangecompressionthegoal is to useretinex to obtainan imagewhich hasdetail in boththedarkandbright regionsof animage. So, for example,if we considerthe top left im-agein Figure1 theaim mightbeto bringout thedetailsintheshadow regionsof this imagewhilst retainingtheinfor-mationcontentof thealreadybright, sunlit regionsof thescene.Thegoalin thispaperis to takethingsastepfurtherandto useretinex to completelyremovetheshadowsfromtheimage.

Shadow removal is interestingfrom perspectivesdiffer-ent thanjust the goal of creatinga morepleasingimage.For exampleit is alsoan importantproblemin computervision sincein this caseshadows canoften confoundal-gorithmsdesignedto solve othervision taskssuchasim-agesegmentationor the locatingand trackingof objectsin a scene.In addition,thereis increasinginterestin theproblemof scenerelighting- takinga scenecapturedun-der someviewing configurationand re-renderingit withdifferent lighting conditions,a problemwhich is on theboundarybetweenvision andgraphics.Locatingandre-moving shadows is an importantstepin solving this typeof problemtoo.

When thinking abouthow we might remove shadowsfrom an imageit is importantthat we considerwhich oftheseapplicationswe aremostinterestedin sincethis canhave a bearingon what restrictionswe placeon how wesolve the problem. For exampleif our goal is imageen-hancementit mightbereasonableto developaninteractiveprocesswhichis ableto removeshadowsgivensomeguid-anceby a user. What is importantis that this processre-ally doesresultin anenhanced(or morepleasing)image.


In thecontext of computervision whereour aim is to ex-tractmeaningfulinformationfrom animage,theaestheticnatureof the imageis not so importantprovided that theresultingimageis shadow freeandcontainsthesalientin-formationof theoriginal image.But in thiscaseit is likelyto beimportantthat theshadow free imageis obtainedbyan automaticprocess.Finally, if our goal is to re-light ascenethenit it is likely thatwewantbothapleasingimageandanautomaticprocessfor obtainingit.

In the context of this paperwe begin by consideringshadow removal from animageenhancementperspective.Weshow how theRetinex theorycanbeextendedto incor-porateshadow removal andsetout the conditionsneces-saryfor thisapproachto work. We demonstratethatwhentheseconditionsaremetretinex basedshadow removalcanwork well. The resultsare twofold: a theoryas to howretinex canbeusedto removeshadowsfrom imagesandaninteractive tool (currentlywritten in MATLAB, but whichcould easilybe incorporatedinto existing imageprocess-ing software) for performingthe shadow removal. In asecondpart to thepaperwe considerhow theprocesscanbeautomated.Our approachherebuilds onprior work [1]whichshowedthatundercertainconditionsit is possibletoobtainilluminationinvariantinformationateachandeverypixel in an image. We show how this informationcanbeincorporatedinto theretinex processandexploretheeffec-tivenessof themethodfor automaticshadow removal.

We begin thepaperwith anintroductionto theRetinexTheoryin Section2, outliningthegeneralprinciplesof thetheory. We thenshow, in Section3, how the theorycanbeextendedto incorporateshadow removal. In Section4we extendthemethodfurtherandoutlinea procedureforautomatingtheshadow removalprocess.Theimplementa-tion detailsof theretinex algorithmhave beenthesubjectof muchdebatesinceLand publishedhis first paper. Wediscussthe majorstrandsof variationin the differentim-plementationsin Section5 andsetouttheprecisedetailsofour own retinex implementationtogetherwith thereasonsasto why we choseit. Finally we give somepreliminaryresultsof ourmethodin Section6.

2. The Retinex Algorithm

The retinex algorithm hasappearedin a numberof dif-ferent incarnations[2, 4, 5, 12, 8, 9, 7] sinceLand firstproposedhis theory. Theseincarnationsarequitedifferentbothin termsof implementationdetailsandtheresultstheyproducebut theunderlyingcomputationthey areperform-ing is essentiallythesamein all cases.For thepurposesofexplaining this underlyingcomputationwe introducethetheoryapieceata time,andnotstrictly in a form in whichit was initially proposed.We assumefirst that an imageis a 2-dimensionalarrayof pixel valuesandthat at each

pixel wehaveatriplet of responseswhichwedenoteRGB.TheseRGBresponsesmight representtheresponsesof thelight sensitiveconecellsat theretina(asin Land’soriginalwork) or moregenerallytheresponsesof any threesensorimagingdevice suchas a digital camera. Let us furtherconsiderthe responsesof eachsensorseparately:that iswe treatan imageasthreeindependent2-d arraysof pix-els.At eachpixel in eachbandor channelwehaveasinglemeasurecorrespondingto the red (R), green(G), or blue(B) sensorresponsesof theimagingdevice.

Land proposedthat rather than our perceptionbeingbaseddirectly on thesesensorresponsesit wasbasedona relative response– a relative measureof brightnessin asinglechannel– whichhecalledlightness. Hefurtherpro-posedthatthe lightnessvalueof a pixel wascomputedbyaseriesof comparisonsof thepixel’s intensitywith thatofmany otherpixels in thescene.In Land’s theorya pixel’slightnessvalueis computedby taking the averageof thepixel’s ratio to many other pixels in the image. To im-plementthis lightnesscomputationLandproposeda pathbasedapproach.Supposewestartatarandompixel whichwedenoteA andfollow a randompath(asequenceof pix-els) to an endpixel Z. Along the way we will visit a se-quenceof pixelswhichwedenoteB, C, D, andsoon. Now,atpixel B wecancomputetheratioof B to A by B

A andwecanstorethis valueat B. Moving to pixel C, we cancom-putetheratio of C to A by takingtheproductof thevaluewehavejuststoredatB: B

A andtheratioof C to B: CB . That

is:BA� C

B� C

A(1)

We storethis value at C and move on to the next pixelin the path. At the endof a pathwe have computedtheratio betweenthestartpixel andeachpixel alongthepathandhave storedthis ratio at the correspondingpixel. Werepeatthis processfor many differentpathswith differentstartandendpoints.Eachtime we visit a pixel we keeparecordof its ratio to the startingpixel. At the endof theprocesswe will have a recordof a pixel’s ratio to manydifferentpixelsandto obtainan estimateof the lightnessatapixel weaverageall theseratios.

Thus,thekey elementsin theretinex computationaretotakearatiofollowedby aproductalongarandompathandto averagetheresultsovermany differentpaths.For com-putationalefficiency thealgorithmis usuallyimplementedin log spacein which ratiosbecomedifferencesandprod-uctsadditions.We canthenexpressthelightnessvalueatagivenpixel j as:

L jR� 1

N

N

∑i � 1

Λi � j (2)

whereΛi � j representsthelightnesscomputedalonga path


�beginningatpixel i andpassingthroughpixel j:

Λi � j � ∑k �� k j

log � Rk� 1 �� log � Rk �� (3)

and N denotesthe numberof pathswhich passthroughpixel j. Thisprocedureis appliedindependentlyto eachofthethreeimagechannelsandthusatriplet of lightnessval-ues: � LR � LG � LB

areobtainedat eachpixel in the image.Land proposedthat it was theselightnesstriplets ratherthantheraw coneresponses,thatarethebasisof ourcolourperception.

A completedefinitionof the retinex algorithmrequiresthat we specify certainparameters:path length,numberof paths,and how a path is calculated,for example. Ithasbeenshown [13] that the choiceof theseparameterscanhave a significanteffect on thealgorithm’s output. Inaddition someversionsof the algorithm supplementtheratio andproductoperationswith two further operations:a thresholdingstep and a resetstep. The thresholdingstepis discussedin the next sectionwherewe introduceour algorithmfor shadow removal. We discussthe resetoperationin Section5 when we give more preciseim-plementationdetailsof the retinex algorithm. Other au-thors [7, 8, 12] have proposedquite different implemen-tations of the Retinex theory and we discussthosetoowhen we describeour own algorithm implementationinSection5.

Finally, an equally importantquestionis what exactlyis beingcalculatedby the retinex algorithm,that is, howis the output relatedto the input? One interpretationofthealgorithmis asa methodby whicha visualsystemcanaccountfor thecolour (spectralcontent)of theprevailingillumination. Loosely, thecomputationnormalisessensorresponsessoasto makethemindependentof theillumina-tion coloursothattheresultinglightnessvaluesarestableacrossachangeof illumination.Theextenttowhichtheal-gorithmsucceedsin thistaskhasbeenstudiedby anumberof authors[6, 13, 9] without definiteconclusions.Otherauthors[8, 12] have proposedthat retinex canbeusedasmethodof compressingthe dynamicrangeof an image,resulting,it is argued,in a morepleasingor enhancedver-sionof the image.Still otherauthorshave arguedthatthealgorithmachievesbothendsat thesametime. In thecon-text of this paperwe areinterestedprimarily in the imageenhancementpropertiesof theapproach.

3. Retinex and Shadow Removal

Landincludedathresholdingstepin hisoriginalalgorithmwhichwasdesignedto removetheeffectsof anilluminantwhoseintensityis varyingacrosstheextentof a scene.Tounderstandhow this works consideragainthatwe followthe path startingat pixel A and finishing at pixel Z. As

we move from A to Z we calculatea sequenceof ratiosbetweenadjacentpixels andby the sequentialprocessofratio andproductalonga pathwe obtainthe ratio of thestart pixel A, to all pixels along the path. If the pixelsarepart of the samesurface,thentheir ratio will be one.However, if pixels are from differentsurfacestheir ratiowill besomethingquitedifferentto one.Supposehoweverthat in additionto a changein surfacereflectancealongapath thereis alsoa gradualchangein the intensityof il-luminationalongthepath.This impliesthatneighbouringpixels which are from the samesurfacecan also have aratio differentto one.Landsuggestedthatbecauseillumi-nationtypically changesmoregraduallythandoessurfacereflectance,theeffect of a changingillumination couldberemovedby thresholdingratiossuchthat if a ratio is onlyslightly different to one it is set to one,but ratioswhicharequite different to oneare left unchanged.By settingratioscloseto oneto beone,theresultinglightnessimageis thesameaswould have beenobtainedwerethe illumi-nationconstantacrossthescene.In thelog domaina ratioof onebecomesa log differenceof zeroandthe lightnesscomputationis modifiedthus:

Λi � j � ∑k �� k j

Tlog � Rk� 1 �� log � rk � (4)

whereT � is defined:

T � x �� 0 if � x �� tx otherwise

(5)

Providedathresholdt canbefoundsuchthatnon-zerolog-differencescanbe reliably classifiedasdueto an illumi-nationchangeor a changein surfacereflectancethenthismodifiedretinex algorithmis ableto remove theeffect ofachangingillumination.

Importantlyshadows too arisebecauseof an effectivechangein illumination. This might bejust a changein theintensityof theilluminationbut moregenerallyit canbeachangein both intensityandspectralcompositionof thelight. Considerfor examplean outdoorscenewhich ispartially in shadow, for examplethe scenein the top leftof Figure1. In the non-shadow regionsof this sceneob-jectsarelit by a mixture of direct sunlightandblue sky-light. By contrast,theshadedregionsarelit only by lightfrom thesky andasa consequenceareilluminatedwith alight which is both lessintenseandalsospectrallydiffer-ent to thesunlit region. But, unlike thecasein which wewish to accountfor a spatiallyvaryingilluminant,wecan-not assumethat this changein illumination from sunlightto shadow is a gradualonefor it canbe asabruptasthechangethat occurswhenwe crossthe boundarybetweendifferentsurfaces.Theshadow region in Figure1 is againagoodexample:theshadow causedby thebuilding resultsin a sharpboundarybetweenlight anddark on the grass.


Figure 1: Top left: a colour image containingsunlit andshadowregions. Top right: a binary image definingthe shadowedge.Bottomleft: theimageafter theshadowremoval process.Bottomright: thecorresponding1-d illuminant invariant image.

Thusthresholdingintensitychangeswill not removeanil-luminationchangewhich is dueto a shadow unlesswesetthethresholdvalueto behigh. But in thiscasewewill alsolikely removedifferenceswhicharedueto achangein sur-facereflectanceaswell. However, supposefor themomentthatwe areableto distinguishin animage,thosechangeswhicharedueto achangein illumination.Thatis, supposethatwecanextracttheshadow edgesfrom animage,in theform of a binaryimagewhichwedenoteS� x � y . With thisshadow edgelet usdefinea new retinex ratio computationthus:

Λi � j � ∑k �� k j

Tshadow

log � Rk� 1 �� log � rk �� (6)

whereTshadow is a new thresholdfunctiondefinedas:

Tshadow � xk �� 0 if Sk

� 1xk otherwise

(7)

Equations(6) and(7) defineournew Retinex with ShadowRemoval algorithm. Applying the processto the top leftimagein Figure1 andusingasthedefinitionof theshadowedgethetopright imageof thefigure,werecoverthelowerleft image. It is clear that the effect of the processis togreatlyattenuatetheshadow regionin theimage:thegrassis now of almostuniformcolourandintensitywhereasbe-fore thereweretwo distinct regions,one in sunlightandonein shadow.

To understandwhy themethodworks,considerthatillu-minationchangecanbemodelledasan independentscal-ing of sensorresponsesin eachchannel(a so-calleddiag-onal modelof illumination changeandconsidera single

surfacewhich is partially in shadow. If theresponseto thepartof the surfacenot in shadow is � Rsun� Gsun� Bsun

andthatto theshadowedregionis � Rshad� Gshad� Bshad

thenourmodelof illumination changegivesus thefollowing rela-tionship:

Rshad� αRsun

Gshad� βGsun

Bshad� γBsun

(8)

wherewe notethat this relationshipholds(for fixed α � β �andγ) for any pairof shadow andnon-shadow pixels.Now,if we take ratiosof, for examplered pixel values,withinthe shadow and red pixel valuesoutsidethe shadow theresultingratio is thesamesincethescalefactor(α in thiscase)cancelsout. Soa retinex pathentirely in shadow orentirelyout of shadow givesthesameresultsateachpixelsinceit is merelycalculatingratioswithin achannel.

Problemsarisehoweverwhenapathcrossestheshadowboundarysincein this casewe arecalculatinga ratio of apixel in shadow to a pixel out of shadow andthe ratio isnow different.But in this caseour thresholdoperatorsetsthe ratio to one(zero in log space)becauseit recognisesthat the edgeis dueto a shadow andso the ratio compu-tation is unaffectedby the shadow edge. And, sincetheshadow edgeis theonly thingthatcanaffecttheratiocom-putation,removing its effect implies thata constantvaluewill becalculatedby thealgorithmfor any pixel of a sur-face,regardlessof whatit is lit by. Thisargumentis borneout by therecoveredimagein Figure2b. We notefurtherthata diagonalmodelof illuminationchangeis implicit inthe original argumentthat retinex canbe usedto removetheeffectof thespectralcontentof thesceneilluminant.

4. Automating Shadow Removal

To enableshadow removal in the retinex framework wehave developeda simpletool which allowsa userto inter-actively definea shadow edge(suchasthetop right imagein Figure1) in animage.Thisshadow edgeis theninputtothe modifiedretinex algorithmalongwith the imageandthe imageis processedto producea shadow free image.Wenotealsothatsincein all otherrespectsthealgorithmisidenticalto theoriginal retinex algorithmtheresultingim-agewill inherit theotherenhancements(suchasdynamicrangecompression)that thealgorithmprovides.While asa tool for imageenhancementthis interactive shadow re-moval is adequate,in othercontexts suchascomputervi-sionit is moreimportantthattheprocessis automaticandsowenow introduceamethodto automatetheshadow de-tectionstep.

Ourapproachherebuildsonpreviousresearch[1] whichshowedthatundercertainconditions(specificallyfor nar-rowbandimagesensorsandPlanckian[14] illumination)it is possibleto derive, from a three-bandcolour image,a


1-dimensionalgrey-scaleimagewhich is invariantto theprevailing sceneillumination. Briefly, the methodworksby first transformingRGBvaluesinto 2-dimensionallog-chromaticityco-ordinates:

r � log � R �� log � G b � log � B �� log � G (9)

Importantlyit canbeshown thatwhentherestrictionssetout above aremet theco-ordinates� r � b changein a pre-dictablemannerwhenillumination changes.Specifically� r � b ’s for a singlesurfaceviewed undermany differentlights lie alonga line in the chromaticityspace.Project-ing orthogonallyto this line resultsin a 1-dvaluewhich isinvariantto illumination.

In fact themethodis quiterobustto situationsin whichtherestrictionsarenotmetexactly. For examplethelowerright imagein Figure1 shows the1-d illuminant invariantimagederived from the top left imageof Figure1 whichwastakenwith acamerawith non-narrowbandsensorsun-deroutdoorillumination. Thecrucialfeatureof this imagein the context of the currentwork is that the shadow hasbeenremoved. This is becausea shadow is causedby achangein illumination andthusremoved in the invariantimage.

In the context of retinex and shadow removal we canutilise this invariant imagein the following way. Let usdenoteby Iinv the1-dgrey-scaleimageandby Icol theorig-inal RGBimage.And let ∇Icol and∇Iinv furtherdenotethegradientimagesof theoriginal colourandinvariantgrey-scaleimagesrespectively. We canthenmodify onceagaintheretinex pathcomputationasfollows:

Λi � j � ∑k �� k j

T �shadow

log � Rk� 1 �� log � rk �� (10)

whereT �shadow is a new thresholdfunctiondefinedas:

T �shadow � �� 0 if � ∇Icol � t1 & � ∇Iinv � t2

xk otherwise

(11)

The thresholdfunction definedby Equation(11) makesuseof informationfrom the invariantimage. Specificallyit setsto 1 any ratioswhich arelikely to coincidewith ashadow edgewhereashadow edgeis definedto beanedgewhich is strongin the original colour image(∇Icol � t1)but weakin the invariantimage(∇Iinv � t2). In otherre-spectsthe algorithm retainsthe path basedcomputationof the original retinex algorithm. In the next sectionwediscusssomemorepreciseimplementationdetailsof thealgorithmswehave introduced.

5. Implementation Details

For our implementationof Retinex we choseto followLand’s original pathbasedcomputationsincetheshadow

removal stepfits seamlesslyinto this framework. In ad-dition to thebasiccomputationalstepsof ratio, threshold,andproductwhich form the basisof the algorithmsomeversionsof theretinex algorithmalsoincludea fourth, re-set step. This stepcomesafter the ratio, threshold,andproductandis simply a checkto seewhetherthe currentvaluecomputedalonga pathis greaterthanone(zeroinlogspace).If it is, thenthepathvalueis resettooneandthecomputationcontinuesasbefore. We have experimentedwith versionsof thealgorithmwith andwithoutaresetstepandhavefoundtheresetversionto givemorepleasingim-agesandsowereportresultsfor thatalgorithmhere.Otherauthors[8, 12] havepresentedquitedifferentimplementa-tionsof thealgorithmtherelative meritsof which arebe-yondthescopeof thispaper. Ourreasonfor rejectingtheseapproachesis that it is difficult to incorporateshadow re-moval into theframework,however, wereweableto doso,it is possiblethatbetterperformancemightbeobtained.

An importantfeatureof our implementationis how thepathsaredefined.In thisregardweadoptanapproachfirstsuggestedby Marini et al [9] andusepathsbasedon theconceptof Brownianmotion [15]. A pathis definedwithrespectto a startandendpixel andis specifiedwhenallinterveningpixelsalongthepathareknown. A Brownianpath is determinedby first choosinga point midway be-tweenthestartandendpixel andthenoffsettingthis pointby somerandomfactord. This resultsin two path seg-ments:thestartpixel to theoffsetmidpointandtheoffsetmidpoint to the endpixel. Theprocessis repeatedrecur-sively for eachline segment for somepre-definednum-berof recursions.Theresultingpathis thusa sequenceofstraightline segmentsandtheretinex computationalongapathfollows all pixels alongall line segmentson a path.Path length,andthedisplacementfactord areparametersof the algorithm. For a given imagewe pre-computeasetof paths(theexactnumberbeinga third algorithmpa-rameter)with respectto a startpixel locatedat � 0 � 0 andrandomendpixels. An imageis thenprocessedpixel bypixel in thefollowing way. Thepre-computedpathsareallshiftedso that they start from the currentpixel andeachpathis followed in turn. In this way it canhappenthat apathleavesthe edgeof the imagein which casethe pathcomputationstopsat this point. After all pathshave beenfollowedfor all pixels,the lightnessvaluesstoredat eachpixel areaveragedto computetheoutputimage.Notethattheretinex computationis definedfor a singlechannelim-ageandto processa typical 3-bandcolourimagewemustrepeatthecomputationthreetimes.In doingsoweusethesamepathsin eachcase.

Marini et al arguedthat using Brownian pathsmeantthatmany fewer pathsneededto befollowedthanif pathsweredefinedin a completelyrandommanner. We havealsofound this to be the case,with 20 to 40 pathsgiving


stablesolutionsfor most images. However we note thatMarini etal did notspecifyexactlyhow they definedtheirpathssoourapproachis closeto theirsin spirit thoughnotnecessarilyin exactcomputationaldetail.

Oneotherimplementationdetailrelatesto how weauto-matetheshadow removal process.Thealgorithmdefinedby Equations(10)and(11)will work well providedwecandetermineappropriatethresholdlevels,t1 andt2. However,a gainin computationalefficiency canbeobtainedby pre-computinga shadow edgeimage(a binaryimagelike thatin thetop left of Figure1) andusingtheretinex algorithmdefinedby Equations(6) and (7) to remove the shadow.Theshadow edgeimageis foundby anautomaticprocesswhich wasfirst describedin [16]. Themethodutilisestheilluminant invariant imageanda grey-scaleintensityim-ageof the original colour image. TheSUSAN [17] edgeoperatoris usedto determineedgesin both imagesandtheshadow edgeis determinedby finding strongedgesinthe intensity imagewhich are weak in the invariant im-age.Finally theseedgesarebinarisedandthickenedusinga morphologicalfilter. We pointout thatthisshadow edgedetectionprocedureis far from perfectandso the resultswepresentin thenext sectionarepreliminaryonly.

6. Results

We now presentpreliminary resultsfor both the interac-tive andautomaticretinex shadow removal algorithmon asmall numberof test images. It is importantwhenusingthe retinex algorithmthat attentionis given to the natureof theinputdata[12]. Ideally theinputdatashouldbelin-earlyrelatedto sceneluminanceandshouldhavesufficientbit-depth:typically greaterthan8-bits.Theimageswetestherewereobtainedwith a commerciallyavailableHP912Photosmartdigital camerawhich wasmodifiedsuchthatwewereabletoaccessraw dataunprocessedby thecamerasoftware.This raw datais 12-bit andsatisfiesthelinearitycondition.

It is also important that thought is given as to wherein the imagereproductionpipelinethe retinex processingshouldbeappliedsinceby default thealgorithmwill per-form somedegreeof correctionfor sceneillumination inadditionto dynamicrangecompressionand, in our case,shadow removal. For theseexperimentswechoseto applythealgorithmdirectly to theraw cameradata. In additionwe wishedto avoid the algorithmperformingany illumi-nantcorrectionwhich we achievedby paddingthe imagewith awhiteborderonequarterthesizeof theoriginal im-age.We usedpathlengthsof approximatelythree-quarterstheimagedimensionssothatmostpathscrossedthewhiteborder. As a resulttheoutputimageswerenot colourcor-rected.

For thepurposesof displaywe performeda numberof

subsequentprocessingstepsto the output images. First,we applieda correctionfor thesceneilluminant (usinganilluminant estimatewhich is a robust maximumin eachchannel)andwethenappliedalineartransformto theRGBresponsesto maptheminto the spacedefineby the ISO-RGBprimaries.Finally weapplieda gammacorrectiontothedatausinga valueof 1� 2 � 2 appropriatefor many CRTdevices. Figure2 shows resultsfor two imageswherethe

Figure 2: Original image (top), selectedshadowedges(middle)and output images with shadowremoved (bottom)for two im-ages.

shadow edgeis definedinteractivelyby auser. Thetoprowshows the original un-processedimagesand the shadowedgefor eachareshown in the middle column. Finally,theoutputof theretinex processingis shown in thebottomrow. It is clearthatin bothcasesthealgorithmdoesagoodjob of removing the shadow. Theseresultsaretypical ofwhat canbe achieved when the shadow edgeis selectedinteractively.

For the sameimageswe usedthe shadow edgedetec-tion schemeoutlined above to derive the shadow edgesautomatically. Figure3 shows resultsfor two images. Itis clear that the shadow edgedetectionis imperfect: foreachcasethe shadow is edgeis found but in addition anumberof spuriousedgesarealso interpretedby the al-gorithmasbeingshadow edges.Usingtheseshadow edgemapsasinputto theretinex algorithmresultsin theimages


shown in the right-handcolumnof Figure5. Onceagainthe shadows areremovedbut the fact that the edgemapsareimperfectresultsin somedegradationof theimagesinareaswheretheprocedurewronglydetectedshadow edges.Nevertheless,theseresultsrepresentencouragingprogressfor a fully automatedshadow removal scheme. In sum-

Figure 3: Automaticallydeterminedshadowedge maps(left) to-getherwith shadowremovedimages(right) for two images.

mary, we have presentedin this papera schemefor re-moving shadows from images.Themethodbuilds on theRetinex algorithmwhich haspreviously beenusedfor anumberof different imageenhancementtasks. We haveprovided a simpleextensionto the methodwhich grace-fully incorporatesshadow removal into theretinex frame-work andwehaveshown thatwhenaccurateknowledgeofthe locationof shadow edgesis known, the methoddoesan excellent job of removing the shadows. In additionwehaveoutlineda methodfor automaticallydetectingtheshadow edgesandhave demonstratedpreliminaryresultswhich show that this automatedprocedureis a promisingone. It remainsto furtherexploretheimageenhancementpropertiesof the procedure,both within the frameworkpresentedhereandalsoin the framework of otherretinexalgorithms,andto further improve the automaticshadowdetection.

References

[1] G.D. FinlaysonandS.D. Hordley. Color constancyat a pixel. J. Opt.Soc.Am.A, 18(2):253–264,2001.Also, UK Patentapplicationno. 0000682.5.Underreview, British PatentOffice.

[2] EdwinH. Land.TheRetinex Theoryof ColorVision.ScientificAmerican, pages108–129,1977.

[3] Edwin H. Land. Lightnessandretinex theory. Jour-nal of the Optical Societyof America,A, 61:1–11,1971.

[4] Edwin H. Land. Recentadvancesin retinex the-ory andsomeimplicationsfor corticalcomputations.Proc.Natl. Acad.Sci.USA, 80:5163–5169,1983.

[5] Edwin H. Land. Recentadvancesin retinex theory.VisionResearch, 26(1):7–21,1986.

[6] SuzanneP. McKee John J. McCann and ThomasH. Taylor. Quantitative Studiesin Retinex Theory.VisionResearch, 16:445–458,1976.

[7] JonathanFrankleand JohnMcCann. Method andapparatusfor lightnessimaging,May 1983. UnitedStatesPatentNo. 4,384,336.

[8] D. J.Jobson,Z. Rahman,andG.A.Woodell.A multi-scaleretinex for bridgingthegapbetweencolourim-agesand the humanobservation of scenes. IEEETransactionson Image Processing:SpecialIssueonColourProcessing, July 1997.

[9] D. Marini andA. Rizzi. A computationalapproachto color adaptationeffects. Image and Vision Com-puting, 18:1005–1014,2000.

[10] B Funt and K Barnard. Investigationsinto Multi-Scaleretinex (MSR), pages17–36.1999.

[11] JohnMcCann. Lessonslearnedfrom mondriansap-pliedto realimagesandcolorgamuts.In TheSeventhColor Imaging Conference, Color Science, Systems,andApplications, pages1–8,November1999.

[12] Brian Funt, Florian Ciurea, and John McCann.Retinex in matlab. In TheEighthColor ImagingCon-ference, Color Science, Systems,and Applications,November2000.

[13] David H. BrainardandBrianA. Wandell.Analysisoftheretinex theoryof colorvision. Journalof theOp-tical Societyof America,A, 3(10):1651–1661, 1986.

[14] G. WyszeckiandW.S. Stiles. Color Science:Con-ceptsandMethods,QuantitativeDataandFormulas.New York:Wiley, 2ndedition,1982.

[15] D. Saupe.Algorithmsfor RandomFractals. Springer,1988.

[16] G.D. Finlayson,Hordley S.D.,andDrew M. S. Re-moving shadows from images. In ECCV02, pages823–836,2002.

[17] S.M.SmithandJ.M.Brady. Susan— anew approachto low level imageprocessing.InternationalJournalof ComputerVision, 23(1):45–78,May 1997.

Documents

Removing Shadows From Images using Retinex · 2006-03-29 · AppearedInProc.10thColourImagingConference,Scottsdale,AZ2002 1 Removing Shadows From Images using Retinex G. D. Finlayson,