Embed Size (px)
Citation preview
Shape from Shading and Shape from Shading and TextureTexture
Lambertian Reflectance ModelLambertian Reflectance Model
• Diffuse surfaces appear equally brightDiffuse surfaces appear equally brightfrom all directionsfrom all directions
• For point illumination, brightness For point illumination, brightness proportional to cos proportional to cos
Lambertian Reflectance ModelLambertian Reflectance Model
• Therefore, for a constant-colored object Therefore, for a constant-colored object with distant illumination, we can writewith distant illumination, we can write
EE = = LL llnnEE = observed brightness = observed brightnessLL = brightness of light source = brightness of light source = reflectance (albedo) of surface = reflectance (albedo) of surfacell = direction to light source = direction to light sourcenn = surface normal = surface normal
Shape from ShadingShape from Shading
• The above equation contains some The above equation contains some information about shape, and in some information about shape, and in some cases is enough to recover shape cases is enough to recover shape completely (in theory)completely (in theory)if L, if L, and l are known and l are known
• Similar to integration (surface normal is Similar to integration (surface normal is like a derivative), but only know a part of like a derivative), but only know a part of derivativederivative
• Have to assume surface continuityHave to assume surface continuity
Shape from ShadingShape from Shading
• Assume surface is given by Z(x,y)Assume surface is given by Z(x,y)
• LetLet
• In this case, surface normal isIn this case, surface normal is
11
122
q
p
qpn
11
122
q
p
qpn
y
Zq
x
Zp
,y
Zq
x
Zp
,
Shape from ShadingShape from Shading
• So, writeSo, write
• Discretize: end up with one equation Discretize: end up with one equation per pixelper pixel
• But this is But this is pp equations in 2 equations in 2pp unknowns… unknowns…
11 22q
p
lllqp
LE zyx
11 22q
p
lllqp
LE zyx
Shape from ShadingShape from Shading
• Integrability constraint:Integrability constraint:
• Wind up with system of 2Wind up with system of 2pp (nonlinear) (nonlinear) differential equationsdifferential equations
• No solution in presence of noise or No solution in presence of noise or depth discontinuitiesdepth discontinuities
x
q
y
p
xy
Z
yx
Z
22
x
q
y
p
xy
Z
yx
Z
22
Estimating Illumination and Estimating Illumination and AlbedoAlbedo
• Need to know surface reflectance and Need to know surface reflectance and Illumination brightness and directionIllumination brightness and direction
• In general, can’t compute from single imageIn general, can’t compute from single image
• Certain assumptions permit estimating theseCertain assumptions permit estimating these– Assume uniform distribution of normals, look at Assume uniform distribution of normals, look at
distribution of intensities in imagedistribution of intensities in image
– Insert known reference object into imageInsert known reference object into image
– Slightly specular object: estimate lighting from Slightly specular object: estimate lighting from specular highlights, then discard pixels in specular highlights, then discard pixels in highlightshighlights
Variational Shape from ShadingVariational Shape from Shading
• Approach: energy minimizationApproach: energy minimization
• Given observed E(x,y), find shape Z(x,y)Given observed E(x,y), find shape Z(x,y)that minimizes energythat minimizes energy
• Regularization: minimize combination of Regularization: minimize combination of disparity w. data, surface curvaturedisparity w. data, surface curvature
dydxqqppyxLyxE yxyx22222),(),( nlE dydxqqppyxLyxE yxyx22222),(),( nlE
Variational Shape from ShadingVariational Shape from Shading
• Solve by techniques from calculus of Solve by techniques from calculus of variationsvariations
• Use Euler-Lagrange equations to get a Use Euler-Lagrange equations to get a PDE, solve numericallyPDE, solve numerically– Unlike with snakes, “greedy” methods tendUnlike with snakes, “greedy” methods tend
not to work wellnot to work well
0
0
yx
yx
qE
qE
qE
pE
pE
pE
dy
d
dx
d
dy
d
dx
d
0
0
yx
yx
qE
qE
qE
pE
pE
pE
dy
d
dx
d
dy
d
dx
d
Enforcing IntegrabilityEnforcing Integrability
• Let Let ffZZ be the Fourier transform of be the Fourier transform of ZZ,,
ffpp and and ffqq be Fourier transforms of be Fourier transforms of pp and and
• ThenThen
• For nonintegrable For nonintegrable pp and and qq these aren’t these aren’t equalequal
y
q
x
pZ i
f
i
ff
y
q
x
pZ i
f
i
ff
Enforcing IntegrabilityEnforcing Integrability
• ConstructConstruct
and recomputeand recompute
• The new The new p’p’ and and q’q’ are the integrable are the integrable equations equations closestclosest to the original to the original pp and and qq
22yx
qypxZ i
fff
22yx
qypxZ i
fff
ZyqZxp fiffif , ZyqZxp fiffif ,
Difficulties with Shape from Difficulties with Shape from ShadingShading
• Robust estimation of L, Robust estimation of L, , l?, l?
• ShadowsShadows
• Non-Lambertian surfacesNon-Lambertian surfaces
• More than 1 light, or “diffuse More than 1 light, or “diffuse illumination”illumination”
• InterreflectionsInterreflections
Shape from Shading ResultsShape from Shading Results
[Trucco & Verri][Trucco & Verri]
Shape from Shading ResultsShape from Shading Results
Active Shape from ShadingActive Shape from Shading
• Idea: several (user-controlled) light Idea: several (user-controlled) light sourcessources
• More dataMore data– Allows determining surface normal directlyAllows determining surface normal directly
– Allows spatially-varying reflectanceAllows spatially-varying reflectance
– Redundant measurements: discard shadows Redundant measurements: discard shadows and specular highlightsand specular highlights
• Often called “photometric stereo”Often called “photometric stereo”
Photometric Stereo SetupPhotometric Stereo Setup
[Rushmeier et al., 1997][Rushmeier et al., 1997]
Photometric Stereo MathPhotometric Stereo Math
• For each point For each point pp, can write, can write
• Constant Constant incorporates light source incorporates light source brightness, camera sensitivity, etc.brightness, camera sensitivity, etc.
3,
2,
1,
,
,
,
,3,3,3
,2,2,2
,1,1,1
p
p
p
zp
yp
xp
zyx
zyx
zyx
p
E
E
E
n
n
n
lll
lll
lll
3,
2,
1,
,
,
,
,3,3,3
,2,2,2
,1,1,1
p
p
p
zp
yp
xp
zyx
zyx
zyx
p
E
E
E
n
n
n
lll
lll
lll
Photometric Stereo MathPhotometric Stereo Math
• Solving above equation gives (Solving above equation gives (//nn
• n must be unit-length n must be unit-length uniquely uniquely determineddetermined
• Determine Determine up to global constant up to global constant
• With more than 3 light sources:With more than 3 light sources:– Discard highest and lowest measurementsDiscard highest and lowest measurements
– If still more, solve by least squaresIf still more, solve by least squares
Photometric Stereo ResultsPhotometric Stereo Results
[Rushmeier et al., 1997][Rushmeier et al., 1997]
InputInputimagesimages
Recovered normals (re-lit)Recovered normals (re-lit)
Recovered colorRecovered color
Helmholtz StereopsisHelmholtz Stereopsis
• Based on Helmholtz reciprocity: surface Based on Helmholtz reciprocity: surface reflectance is the same under reflectance is the same under interchange of light, viewerinterchange of light, viewer
• So, take pairs of observations w. viewer, So, take pairs of observations w. viewer, light interchangedlight interchanged
• Ratio of the observations in a pair is Ratio of the observations in a pair is independent of surface materialindependent of surface material
Helmholtz StereopsisHelmholtz Stereopsis
[Zickler, Belhumeur, & [Zickler, Belhumeur, & Kriegman]Kriegman]
Helmholtz StereopsisHelmholtz Stereopsis
TextureTexture
• Texture: repeated pattern on a surfaceTexture: repeated pattern on a surface
• Elements (“textons”) either identical or Elements (“textons”) either identical or come from some statistical distributioncome from some statistical distribution
• Shape from texture comes from looking Shape from texture comes from looking at deformation of individual textons or at deformation of individual textons or from distribution of textons on a surfacefrom distribution of textons on a surface
Shape from TextureShape from Texture
• Much the same as shape from shading, but Much the same as shape from shading, but have more informationhave more information– Foreshortening: gives surface normal (not just Foreshortening: gives surface normal (not just
one component, as in shape from shading)one component, as in shape from shading)
– Perspective distortion: gives information about Perspective distortion: gives information about depth directlydepth directly
• Sparse depth information (only at textons)Sparse depth information (only at textons)– About the same as shape from shading, About the same as shape from shading,
because of smoothness term in energy eqn.because of smoothness term in energy eqn.
Shape from Texture ResultsShape from Texture Results
[Forsyth][Forsyth]
