Intrinsic Image Separation Using Weighted Map and Correction

Using MRFs

Robotics LabDepartment of Computer Science and

Information EngineeringNational Cheng Kung University

謝松憲、方志偉、王德勛、朱健宏、連震杰

MotivationWhy separating Shading images and

Reflectance images?Reflectance images are more appropriate for

pattern recognition, object detection and scene interpretation.

Shading images can be used for shading analysis, illumination assessment.

1. Introduction(1/2)

22

1.Introduction(2/2)

33

=

Input Shading Reflectance

I(x,y) = S(x,y) R(x,y)

H.G. Barrow and J.M. Tenenbaum, “Recovering Intrinsic Scene Characteristics from Images,” Computer Vision System, A. Hanson and E. Riseman, eds., pp. 3-26. Academic Press, 1978.

log I(x,y) = log S(x,y) log R(x,y)

Our approachClassify image derivatives

Each derivative is caused either by shading or reflectance ,but not both.

Derivatives caused by reflectance changes have a greater magnitude than those caused by shading.

Assumption

44Y. Weiss, “Deriving Intrinsic Images from Image Sequences,” Proc. Int’l Conf. Computer Vision, 2001.

2.System Flowchart

55

Input color Image Input color Image

Logarithmic Logarithmic TransformationTransformation

Derivative Component Derivative Component ImageImage

Intrinsic Derivative Intrinsic Derivative Component CreationComponent Creation

DeconvolutiDeconvolutionon

Logarithmic Intrinsic Logarithmic Intrinsic Component ImageComponent Image

Exponential Exponential transformtransform

Intrinsic Intrinsic ImagesImages

Module 1：Intrinsic Derivative

Component Creation

Color Domain Transformation Color Domain Transformation into LUM-RG-BYinto LUM-RG-BY

Derivative Component ImageDerivative Component Image

Weighted MapWeighted Map

Derivative Component Derivative Component Correction Based on ProbabilityCorrection Based on Probability

cc

cc

Module 3 ： Misclassification Correction Using

MRFs and Loopy Belief Propagation

Module 4 ： Intrinsic Image Recovery

Classification Classification Using Weighted Using Weighted MapMap

Misclassification Misclassification Correction Correction

Module 2： Weighted-Map Creation and

Derivative Component Classification

2.1 Module 1 ： Intrinsic Derivative Component Creation(1/2)

66

''' loglog)log(log iiiiiiii RSRSRSII bgri ,,

Logarithmic transformation

log R log G log B

R G B

2.1 Module 1 ： Intrinsic Derivative Component Creation(2/2)

77

tyx f f 1] 1- 0[ , 1] 1- 0[

Horizontal derivative xii

x fII ''

Vertical derivative yii

y fII ''

Derivative convolution

'rxI 'g

xI 'bxI

'ryI 'g

yI 'byI

2.2 Intrinsic Derivative Component Creation

88

bgriyxIyxSS

yxIyxRRif

ix

ix

ix

ix ,, ,

),(),( ,

),(),( ,

''

''

i.e. for horizontal direction 'i

xR

'ixS

Derivative Derivative componentscomponents

Module 1.

Module 2&Module 3

Intrinsic Derivative Intrinsic Derivative Component CreationComponent Creation

ClassifiedClassifiedresultresult

Deconvolution

Exponential transform

Composition

2.3 Module 4 ：Intrinsic Image Recovery Process

99

)]()[(

)]()[('''

'''

iy

ry

ix

rx

i

iy

ry

ix

rx

i

RfRfgR

SfSfgS

where

)]()[( ),()( yr

yxr

xjrj ffffgpfpf

bgriRRSS iiii ,, ),exp( ),exp( ''

Y. Weiss, “Deriving Intrinsic Images from Image Sequences,” Proc. Int’l Conf. Computer Vision, 2001.

1] 1- 0[f

0] 1- 1[rf

) , ,(

) , ,(bgr

bgr

RRRR

SSSS

LUM,RG,BY color space

3.1 Module 2:Part A ：Color Domain Transformation

1010

LUMS

LUMSBY

LUM

MLRG

MLLUM

*5.0

*5.0

LUM RG BYShading component

Kingdom, F. A. A., Rangwala, S.& Hammmamji, “Chromatic Properties of the Color Shading Effect,” Vision Research, 45, 1425-1437, 2005

Shading component ： only in LUM image plane!!

Reflectance component ： in all three image planes

0.3811 0.5783 0.0402

0.1967 0.7244 0.0782

0.0241 0.1288 0.8444

L R

M G

S B

3.2 Module 2: Part B ：Filter Convolution

1111

xLUM xRG xBY

yLUM yRG yBY

tyx f f 1] 1- 0[ , 1] 1- 0[

Reflectance-related mapIdea ： extract reflectance component

Weighted-map

3.3 Module 2: Part C ：Weighted-Map Classification(1/2)

1212

)) ,( ,) ,(max() ,(

)) ,( ,) ,(max() ,(

yxBYyxRGyxM

yxBYyxRGyxM

yyy

xxx

( , ) ( , ) ( , )

( , ) ( , ) ( , )

x x x

y y y

W x y LUM x y M x y

W x y LUM x y M x y

Threshold & classification

3.3 Module 2: Part C ：Weighted-Map Classification(2/2)

1313

Shading otherwise,

eReflectanc , if

yy tW；

Shading otherwise,

eReflectanc , if xx tW

xW yW

Intrinsic images

3.4 Experimental Results(1/2)

1414

Input image I

Shading image S Reflectance image R

B * 0.1140 +G * 0.5870 + R * 0.2989Gray value

Intrinsic images

3.4 Experimental Results(2/2)

1515

Input image I

Shading image S Reflectance image R

Problem:There are still some misclassifications after

using weighted-map method.

4. Misclassification

1616

Misclassifications!Misclassifications!Conclusion:

Most derivatives on each edge are correctly classified as reflectance.

A small number of pixels on the same edge may be misclassified as shading.

Step1:

4. Modeling Using Markov Random Fields(1/3)

1717

RR SS RR

RR

RR RR RR RR

RR RR

RR

RR

RR

SS

SS

SS

SS RR

RR RR RR SS

SS

SS RR RR RR

RR

RR RR

=

1, pixel derivative is classified as Reflectance

= 0, pixel derivative is classified as Shading

, i ii

ix y

= 11

00 11 11

11 11 11 11

11 11

11

11

11

00

00

00

00 11

11 11 11 00

00

00 11 11 11

11

11 11

where xi represents the hidden node state and yi represents the observation node state at pixel i.

'gxR

'gxS

Step2: Initialize MRFs and define joint compatibility function.

4. Modeling Using Markov Random Fields(2/3)

1818

11 00 11

11

11 11 11 11

11 11

11

11

11

00

00

00

00 11

11 11 11 00

00

00 11 11 11

11

11 11

Observation nodeObservation node

hidden nodehidden node

0.7, if and are in same state( )=

0.3, if and are in different state

i i

i ii

y

y

xx

1111

1111

1111

1111

1100

00

1100

00

1111

1100

1111

0011

00

1111

00

00

1111

00

0.7, if and are in same state( , )=

0.3, if and are in different state

i j

i ji j

x xx x

1111

1111

1111

1111

1100

00

1100

00

1111

1100

1111

0011

00

1111

00

00

1111

00

1111

1111

1111

1111

1111

11

1100

00

1111

1111

1111

0011

11

1111

00

1111

00

11

Step 3:Maximize objective function P by adjusting all hidden node states.

4. Modeling Using Markov Random Fields(3/3)

1919

( , )

1( , ) ( )

i j i

P i j iZ

Original MRFsOriginal MRFs

MRFs after MRFs after maximizing maximizing PP

11 00 11

11

11 11 00

00

00

00 11 11 11 11 11

11 11 11

11

11 11 11

11

11

11 11 11 11 11 11

Original Original ClassificationClassification

Misclassification Misclassification CorrectionCorrection

Adjusting all hidden node states is

time consuming.Use Loopy Belief Propagation to get a approximation solution.

5. Experimental Results(1/3)

2020

No Misclassification No Misclassification CorrectionCorrection

Misclassification Misclassification CorrectionCorrection

5. Experimental Results(2/3)

2121

Input image I

Shading image S Reflectance image R

5. Experimental Results (3/3)

2222M.F. Tappen, W.T. Freeman, and E.H. Adelson, “Recovering Intrinsic Images from a Single Image,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 27, No. 9, pp. 1459-1472, 2005.

Input image Our Result Tappen’s Result

Input image Our Result Tappen’s Result

Thanks for your attention~

J.S. Yedidia, W.T. Freeman, and Y. Weiss, “Understanding Belief Propagation and its Generalizations,” MITSUBISHI Electric Research Lab, TR-2001-22, 2002

Appendix

2424

The GoalA image is composed of two parts, called

Shading and Reflectance images. We proposed a method for separating Shading and Reflectance images given a single input image.

DefinitionWhat are Shading and Reflectance?

Reflectance: Remain constant under different illumination conditions.

Shading: Vary from different illumination conditions.

1.Introduction(1/3)

2525H.G. Barrow and J.M. Tenenbaum, “Recovering Intrinsic Scene Characteristics from Images,” Computer Vision System, A. Hanson and E. Riseman, eds., pp. 3-26. Academic Press, 1978.

3. Weighted-Map Method ： Flowchart

2626

Input image I

LMS ImageLMS Image

LUM RG BY

Derivative filters convolution Derivative filters convolution yx ff ,

xLUM yLUM xRG yRG xBY yBY

Part APart A ：：Color Domain Color Domain TransformatioTransformatio

nn

Part BPart B：：Filter Filter

Convolution Convolution

Part CPart C：：Weighted-Map Weighted-Map ClassificationClassification Weighted-map Weighted-map

xWWeighted- map Weighted- map

yWThresholdThreshold

&&ClassificationClassification

Reflectance-related Reflectance-related Map Map yM

Reflectance-related Reflectance-related MapMap xM| || | | || |

|max| |max|

**