Shadow removal algorithms

Shadow removal seminarPavel Knur

Deriving intrinsic images from image sequences

Yair WeissJuly 2001.

History

• “intrinsic images” by Barrow and Tenenbaum , 1978

Constraints

• Fixed viewpoint• Works only for static objects• Cast shadows

Classic ill-posed problem

•Denote– the input image– the reflectance image– the illumination image

Number of Unknowns is twice the number of equations.

),( yxR),( yxI

),( yxL

),(),(),( yxRyxLyxI

The problem

Given a sequence of T imagesin which reflectance is constant over the time and only the illuminationchanges, can we solve for a singlereflectance image and T illumination images ?

Still completely ill-posed : at every pixel there are T equations and T+1 unknowns.

)},,({1

tyxIT

t

)},,({1

tyxLT

t

Maximum-likelihood estimation

• Log domain :

),(),(

),(),(

),(),(

log

log

log

yxlyxL

yxryxR

yxiyxI

),,(),(),,( tyxlyxrtyxi

Assumptions

When derivative filters are applied to natural images, the filter outputs tend to be sparse.

Laplacian distribution

Can be well fit by laplacian distribution

xZ exP 1)(

Claim 1

Denote :• N filters – • Filter outputs – • Filtered reflectance image –

ML estimation of filtered reflectance image

is given by

}{ nf

nn ftyxityxo ),,(),,(

nn frr

nr̂

),,(ˆ tyxomedianr ntn

Estimated reflectance function

Recover ML estimation of r

is reversed filter of

nn rrf ˆˆ

)ˆ(ˆ n

nrn rfgr

rnf nf

)(n

nrn ffg

ML estimation algorithm

ML estimation algorithm – cont.

• Ones we have estimated ),( yxr

),(),,(),,( yxrtyxityxl

Claim 2

•What if does not have exactly a Laplasian distribution ?

Let

Then estimated filtered reflectance are within with probability at least:

),,( tyxlfn

)),,(( tyxlfPp i

2/

1

)1(T

k

kkT ppk

T

Claim 2 - proof

If more than 50% of the samples ofare within of some value, then by definition of median, the median must be within of that value.

),,( tyxlfn

Example 1

• Einstein image is translated diagonally

• 4 pixels per frame

Example 2

• 64 images with variable lighting from Yale Face Database

Illumination Normalization with Time-Dependent Intrinsic Images for Video SurveillanceY.Matsushita,K.Nishito,K.IkeuchiOct. 2004

Illumination Normalization algorithm

• Preprocessing stage for robust video surveillance.

• Causes– Illumination conditions– Weather conditions– Large buildings and trees

• Goal– To “normalize” the input image

sequence in terms of incident lighting.

Constraints

• Fixed viewpoint• Works only for static objects• Cast shadows

Background images

• Remove moving objects from the input image sequence

Input images

Background images

Off-line

Estimation of Intrinsic Images

Denote• input image• time-varying reflectance image• time-varying illumination image• reflectance image estimated by ML• illumination image estimated by

ML

• Filters

• Log domain

Input images

Background images

Off-line

Estimation of Intrinsic Images

),,( tyxR

),,( tyxL

),( yxRw),,( tyxLw

),,( tyxI

),,(),,(),,( tyxRtyxLtyxI

1100 f

Tf 1101

ww lrlri ,,,,

Estimation of Intrinsic Images – cont.

• In Weiss’s original work

• The goal is to find estimation of and

Input images

Background images

Off-line

Estimation of Intrinsic Images

),,(),(ˆ tyxifmedianyxr ntwn

),(ˆ),,(),,(ˆ yxrtyxiftyxl wnnwn

ril

),,( tyxR ),,( tyxL

Estimation of Intrinsic Images – cont.

Basic idea:• Estimate time-varying reflectance

components by canceling the scene texture from initial illumination images

Define:

Input images

Background images

Off-line

Estimation of Intrinsic Images

otherwisetyxl

Tyxriftyxl

wn

wnn ),,,(

),(,0),,(

otherwiseyxr

Tyxriftyxlyxrtyxr

wn

wnwnwnn ),,(

),(),,,(),(),,(

),,(),,(),,(),(),,( tyxltyxrtyxlyxrtyxif nnwnwnn

Estimation of Intrinsic Images – cont.

Finally :

Where : is reversed filter of

Input images

Background images

Off-line

Estimation of Intrinsic Images

nn

rn

nn

rn

lfgtyxl

rfgtyxr

ˆ),,(ˆ

ˆ),,(ˆ

r

nf nf

)(n

nrn ffg

Shadow Removal

Denote - background image - illuminance-invariant image

Input images

Background images

Off-line

Estimation of Intrinsic Images

),,( tyxB

),,(),,(),,( tyxLtyxRtyxB

),,( tyxN

),,(/),,(),,( tyxLtyxBtyxN

Illumination Eigenspace

• PCA – Principle component analysisBasic components -

Input images

Background images

Off-line

Estimation of Intrinsic Images

nsss ,...,, 21

IlluminationEigenspace

Illumination Eigenspace – cont.

• Average is

• P is MxN matrix where– N – number of pixels in illumination

image– M – number of illumination images

• Covariance matrix Q of P is

Input images

Background images

Off-line

Estimation of Intrinsic Images

n ww L

nL

1

IlluminationEigenspace

wwwwww LLLLLLPn ,...,,

21

TPPQ

iii Qee

Direct Estimation of Illumination Images

• Pseudoillumination image

• Direct Estimation is

• Where– F is a projection function onto the j’s

eigenvector

-

Input images

Background images

Off-line

Estimation of Intrinsic Images

),(/),,(* yxRtyxIL w

IlluminationEigenspace

j

wjLw jLFjLFwLiiw

2* ),(),(minargˆ

i

jjw

Direct Estimation of Illumination Images

• Results

Input images

Background images

Off-line

Estimation of Intrinsic Images

IlluminationEigenspace

Shadow interpolation

probability density functioncumulative probability functionshadowed arealit area

mean

optimum threshold value

Input images

Background images

Off-line

Estimation of Intrinsic Images

IlluminationEigenspace

ShadowInterpolation

T

iis ipTP

min

)()(

max

)()(i

Til ipTP

T

iis iipT

min

)()(

max

)()(i

Til iipT

2)()()()(maxarg TTTPTPT lsls

T

opt

)(ip

Ps

l

optT

The whole algorithmInput images

Background images

Off-line

Estimation of Intrinsic Images

IlluminationEigenspace

/

IlluminationImages

Normalization

ShadowInterpolation

Example

Questions ?

References

[1] Y.Weiss,”Deriving Intrinsic Images from Image Sequences”, Proc. Ninth IEEE Int’l Conf. Computer Vision, pp. 68-75, July 2001.

[2] Y.Matsushita,K.Nishito,K.Ikeuchi,“Illumination Normalization with Time-Dependent Intrinsic Images for Video Surveillance”,Oct. 2004.