Doorschot (2001) The combined influence of binocular

Copyright 2001 Psychonomic Society Inc 1038

Perception amp Psychophysics2001 63 (6) 1038-1047

Ecological optics reveals various aspects of a scene ora picture of a scene as potential depth cues or shape cuessuch as texture gradients binocular disparity contourand shading A large number of studies have been carriedout on the role of single cues in human perception In reallife typically many cues are available to an observer si-multaneously so it is important to proceed from the studyof single cues to the case of combined cues This is a com-plicated issue because of the varied nature of the cuesSeveral studies have focused on this subject

In our experiment we addressed the combination ofbinocular disparity and shading as the cause of pictorialrelief These are two cues whose individual effects havebeen well researched In the following paragraphs weargue that these cues provide data that are potentially com-plementary in many respects which makes it interestingto see how they combine Another important reason whythese particular cues have been chosen is that they can bevaried parametrically

First we consider the shading cue We use the termshading to refer to a number of aspects of surface scat-tering all of which may provide potential shape infor-mation For many surfaces surface scattering can be de-composed roughly into a specular component (regularreflection) and a Lambertian component The specularcomponent causes highlights the Lambertian componentleads to radiance variations when the local surface attitudevaries with respect to the major direction of irradiation Inaddition the light source or part of it can be shielded from

the surface by another surface part In the case of a colli-mated source this yields attached and possibly cast shad-ows and when the light source is extended one speaks ofvignetting The transition between cast shadows and vi-gnetting is a gradual one

We would like to point out that the shading cue is notof a global nature because (generally) the irradiating beamis not uniform over the whole scene For instance even ifwe look at a simple object that is illuminated by just onesource the light comes from a different direction for partsthat are illuminated directly as compared with parts thatare in the shadow Furthermore a linear radiance gradientis compatible with any type of constant curvature so it canfor instance be both a concave and a convex object thatcaused an exactly identical gradient Ramachandran (1988)argued that the shading cue is used on the assumptionthat the whole scene an observer sees is irradiated fromonly one direction Even if that is indeed the case shad-ing patterns can still vary tremendously depending on theexact location of the light source At the same time agiven specific shading pattern can be generated by manydifferent shapes So unless an observer has access some-how to source direction and albedo variation the shadingcue is of an ambiguous nature

Second we consider the binocular disparity cue For thevisual system to be able to use this as a cue surfaces needto be revealed by well-localized surface markings (seefor instance Blake Zisserman amp Knowles 1985 Marr1982) Potential features are edges texture and shadingHowever straightforward fusion of smooth edges onsmooth objects like for instance a sphere leads to in-correct disparity because due to self-occlusion the edgein the right image originates from a different part of thesphere than does the edge in the left image (Todd Nor-man Koenderink amp Kappers 1997) Fusion of dihedraledges like those of a cube however leads to correct dis-

This research was supported by the Life Sciences Foundation (SLW)which is subsidized by the Netherlands Organization for Scientific Re-search (NWO) Correspondence concerning this article should be ad-dressed to A M L Kappers Helmholtz Instituut Universiteit UtrechtPrincetonplein 5 3584 CC Utrecht The Netherlands (e-mail amlkappersphysuunl)

The combined influence of binoculardisparity and shading on pictorial shape

PAUL C A DOORSCHOT ASTRID M L KAPPERS and JAN J KOENDERINKUniversiteit Utrecht Utrecht The Netherlands

The combined influence of binocular disparity and shading on pictorial shape was studied Stimuliwere several pairs of stereo photographs of real objects The stereo base was 0 7 or 14 cm and the lo-cation of the light source was varied over three positions (one from about the viewpoint of the cameraone about perpendicular to the line of sight and one in between the two) Therefore in total nine dif-ferent combinations were studied Subjects had to perform surface attitude settings at about 300 posi-tions in the image plane From the settings depth maps were calculated on which a principal compo-nents analysis was performed It was found that three components were enough to account for at least978 of the variance in the data The first component accounted for shape constancy The effects ofthe two cues could be isolated as a linear combination of the other two components The effects of thedisparity and the shading cue variation were found to combine in almost linear fashion

CUE COMBINATION IN SHAPE PERCEPTION 1039

parity By texture we mean variation of the surface albedorather than three-dimensional (3-D) surface roughnessLike dihedral edges texture is well localized and there-fore leads to correct disparity Radiance variation due toattached and cast shadow boundaries also creates possi-bilities for useful binocular matches The specular com-ponent leads to matches whose disparities do not revealthe surface in a simple way because the spatial structureof this component depends on the vantage point

In the present experiment we varied the disparity cueparametrically by varying the stereo base from zero toabout twice the natural stereo base in pairs of stereo pho-tographs When the stereo base is varied three parameterswill be altered the absolute disparity and with it the rel-ative disparity (see Blakemore 1969 for the distinctionbetween the two) and the vergence It is important to notethat a change of any of these parameters will globally af-fect the percept of a scene Collewijn and Erkelens (1990)argued that any role played by vergence-related signals orabsolute disparity in the estimation of distance can onlybe weak Under full cue conditions other cues dominateHowever binocular stereopsis is sensitive to relative dis-parities and relative disparity is an effective cue to rela-tive depth differences From geometrical considerationsit can be concluded that when the stereo base is alteredrelative disparity will signal a differently scaled dispar-ity f ield Therefore the same object viewed with ahigher stereo base might be interpreted as being moreelongated in depth (see also Howard amp Rogers 1995 fora theoretical discussion on stereopsis) On those groundswe expect global scaling effects to occur when the stereobase is varied This expectation is based only on the as-sumption that the visual system presumes a constantstereo base which indeed seems to be the case (Howardamp Rogers 1995)

In the past such scaling effects have indeed been foundin binocular viewing experiments in which the stereobase was varied (Koenderink van Doorn amp Kappers1995) The natural stereo base of the subjects and stereobase zero were used It was found that in the zero dispar-ity condition shapes were globally judged to be flattenedHowever in these experiments unlike in the present ex-periment subjects looked at a real scene instead of pho-tographs Note that in the stereo base zero condition rel-ative disparity signals a disparity field with zero depthso if the observer had based shape judgments on dispar-ity alone shapes would have been judged to be totally flatwhich was not the case

We mentioned that the disparity and the shading cueprovide complementary shape information and dependon different prior assumptions We pointed out that theshading cue is not of a global nature and it also has anambiguous nature In contrast the disparity cue is of aglobal nature When a scene is illuminated by for in-stance two light sources local parts of the scene may beilluminated by only one source However there is no waythat we can for instance have a real scene with one stereobase on one part and another one on another part Anotherdifference is the fact that shading variations depend

mainly on the surface curvature but disparity dependsonly on the distance For instance if we look at a uniformlyilluminated flat slanted surface in isolation shading pro-vides us with no information whatsoever about this slant(or the depth) In that case disparity is still available

We mentioned that shading can help to provide dis-parity information by means of for instance attachedshadow boundaries On the other hand disparity informa-tion can be used to disambiguate shading (eg Blake et al1985) So the two cues can be complementary There-fore it will be very interesting to see whether and if sohow these two cues are combined and whether effects ofdisparity variation depend on the lighting of the scene

Buumllthoff and Mallot (1988) addressed the issue of cueinteraction between disparity and shading on simple ob-jects (ellipsoids) with computer-generated shading Theyfound that the amount of elongation in depth of judgedsurfaces was an accumulation of the contribution of thetwo cues They also found that despite the smooth natureof radiance variation due to the Lambertian componentthis could still be used as a feature for useful binocularstereopsis A major difference between our experiment andthe experiment of Buumllthoff and Mallot is that we use amuch more complex stimulus

Landy Maloney Johnston and Young (1995) reviewedthe literature on cue combination and interaction and pre-sented a formal model of depth cue combination that isconsistent with many experimental results The main pur-pose of this model is not to describe how the visual sys-tem works but rather to guide cue combination experi-ments The model assumes a depth system that is dividedinto different modules for different depth cues In eachmodule a depth map is ldquocomputedrdquo and also the relia-bility of this depth map is estimated This reliability canvary between positions in a scene The various depth esti-mates are combined into a single depth map by a weightedlinear average where the weights take into account theestimated reliability and also possible discrepancies be-tween depth maps By also accounting for these discrep-ancies in the final depth estimate Landy et al (1995)stressed that their model becomes robust which meansthat if the deviation of one cue as compared with theother cues increases from zero it should affect the finaldepth estimate linearly However when the deviations be-come larger and increase beyond the range present in nor-mal scenes the influence of this one discrepant cue is as-sumed to have less and less effect on the percept In themost extreme case a cue can even be vetoed in this modelif the information from this cue is too different from theinformation supplied by other cues

Next we bring up some issues of methodology A com-mon way to investigate the efficacy of a single cue is totry and isolate it from all the others In most cases how-ever one obtains ambiguous results For instance in thecase of shading subjects fail to obtain a clear impressionof depth when other cues are totally absent (eg ErensKappers amp Koenderink 1993) An alternative is to varya single cue parametrically while all others are kept con-stant Such a perturbation analysis has been used success-

1040 DOORSCHOT KAPPERS AND KOENDERINK

fully in the case of shading by for instance Koenderinkvan Doorn Christou and Lappin (1996a 1996b) andTodd Koenderink van Doorn and Kappers (1996) Thismethodology will therefore be applied in the present ex-periment Perturbation studies but of a different kindhave also been used to measure cue combinations or in-teractions (see eg Johnston Cumming amp Parker 1993Landy Maloney amp Young 1991 Young Landy amp Mal-oney 1993) In the latter experiments conflicting cueswere presented and the weights based on the model ofLandy et al (1995) were measured A disadvantage ofcue conflicts is that the visual system is forced to operatein artificial and ambiguous situations and the results donot necessarily apply to natural conditions

Various paradigms address the perception of spatialstructures For instance one may ask for the sign of el-liptic curvature (ie convex or concave Ramachandran1988) for the magnitude of the curvature (Buckley ampFrisby 1993) whether a line is perpendicular to a surface(Stevens amp Brookes 1988) and so forth In such tasksonly simple local shape aspects are to be judged althoughthe result may well depend on the global image We useda very simple task for local slant and tilt (Koenderink vanDoorn amp Kappers 1992) This task has been used exten-sively (eg see Koenderink et al 1996a 1996b Koen-derink et al 1992 1995 Norman Todd amp Phillips 1995Todd et al 1996 Todd et al 1997) The advantages areas follows A vast amount of data of geometrical naturecan be gathered in a short amount of time when desireda global depth map of the investigated shape can be de-rived almost no training is required to perform the taskand even over longer periods of time subjects are con-stant in their settings Furthermore subjects report that

the task comes to them naturally and they are certain oftheir settings This task lends itself very well for pertur-bation studies in more natural complicated scenes

We present results of an experiment in which we ad-dressed the issue of whether and if so how the disparitycue and the shading cue are combined in a natural settingWe varied these cues parametrically In order to presentrealistic shading we avoided computer graphics and usedphotographs Stimuli were plastic human torsos that hadvery intricate shapes (see Figure 1) The model developedby Landy et al (1995) predicts that the cues will combinelinearly If that prediction turns out to be true we wouldexpect to reproduce effects that are known or expectedfrom studies of the cues in isolation local effects for theshading cue and more global effects for the disparity cue

METHOD

StimuliTwo plastic torsos were used as stimuli one representing a male

body (stimulus A) the other a female body (stimulus B) The sur-faces were textured Both objects were photographed in dorsal view(see Figure 1) Pairs of stereo photographs were presented to thesubjects The stereo bases were 0 7 and 14 cm The left photographin these sets was always the same photo for the two stimuli

The objects were photographed under three different illumina-tion conditions (see Figure 2 for a set-up of the photo studio) In allcases the camera was positioned 245 m from the stimulus Weused a light source with a directed diffuse beam (halogen light bulbin umbrella reflector of about 90-cm diameter) which was posi-tioned about 1 m higher than the camera at angles to the line of sightof 176ordm (the front lighting condition) 785ordm (the side lighting con-dition) and 101ordm (the rim lighting condition) and at distances of288 146 and 141 m from the stimulus respectively In Figure 1AB C E F and G the different illumination conditions are shownfor the two stimuli Figures 1C and 1D show the uncrossed fusion

a b a

c d c

e f

g h g

Figure 1 The stimuli used in the experiment In A B and C the photographs for stimulus A are shown for the three different light-ing conditions The photographs in A B and C were taken with the light source positioned about 1 m above the camera at angles of176ordm 785ordm and 101ordm to the line of sight respectively In C and D the (uncrossed) stereo pair of photographs of this last lighting con-dition (rim) is shown with a stereo base of 14 cm A copy of C is also displayed on the right side of D for crossed fusers In E F andG the photographs are shown for the same lighting conditions but now they depict stimulus B The stereo pair displayed in G and Hhas a stereo base of 7 cm Again a copy of G is also displayed on the right side of H for crossed fusers

A B

C D C

E F

G H G


stereogram with a stereo base 14 cm for stimulus A in the rim light-ing condition and Figures 1G and 1H show a stereogram for stimulusB in the rim lighting with stereo base 7 cm Copies of Figures 1Cand 1G are also displayed on the right side of Figures 1D and 1Hrespectively for people who prefer crossed fusion stereograms

So in total there were two objects 3 disparity conditions and 3lighting conditions which makes 18 different conditions in all

Presentation of the StimuliThe experiment was performed on a Quadra 950 Macintosh

Computer with a PowerPC card Two monitors were used one colormonitor (305 3 405 cm) on which the stimuli were presented andone gray-scale monitor (175 3 225 cm) on which an interactionbox was shown This box was used for user interactions and cuingmessages The pictures were scanned with a Hewlett-Packard Scan-jet plus scanner which produced very high quality pictures on themonitor comparable to good postcard-sized photographs (resolu-tion 283 pixels per cm stimulus A 347 3 520 pixels stimulus B359 3 520 pixels) The pictures were viewed through a standardmirror stereoscope with a convergence angle of zero In reality theobjects had a height of 89 cm and the photographs were taken froma distance of 245 m The height of the objects on the monitor was185 cm so to look at them from the correct perspective the view-ing distance should be 55 cm This was accomplished by fixing thestereoscope at the right distance In that case there were no cue con-flicts between the vergence and the disparity cue The room inwhich the experiments were performed was dimly lit so that theoutlines of the monitors were dimly visible

ProcedureOn the object in the left photograph a red (monocular) wire-

frame gauge figure was superimposed which the subject could ma-nipulate (see Koenderink et al 1992) This gauge figure looked

like the orthographic projection of a circle (diameter raquo 8 mm) witha stick perpendicular to the plane of the circle protruding from thecenter (length raquo 4 mm) The task for the subject was to manipulatethis gauge figure until it looked like a circle painted on the surfaceof the object the stick being the outward surface normal In thisway the stick helped to resolve the inherent 180ordm ambiguity in thetilt The gauge figure was presented monocularly to prevent thesubjects from matching local disparities of the object and the gaugefigure The subjects saw the probe attached to the surface Thegauge figures were subsequently presented in random order on thevertices of a triangulation For this purpose the object in the leftphotograph was triangulated with a regular grid Since all left pic-tures were the same only two different triangulations were usedone for stimulus A and one for stimulus B Stimulus A was trian-gulated into 272 vertices stimulus B into 265 The subjects neversaw the triangulation during the experiments An example of thetriangulation for stimulus A is given in Figure 3

In one session which typically took about 1 h the subjects werepresented with 1 of the 18 conditions and had to adjust the gaugethree times on all the vertices in a randomized order For 4 selectedconditions in eight extra sessions the subjects performed an addi-tional six trials per vertex These 4 conditions were stimulus Awith stereo base 0 cm photographed with both the front and the rimlighting condition and stimulus A with stereo base 14 cm alsowith both front and rim condition Measuring every condition ninetimes would have been too time consuming but we did want to geta better impression of the statistics than one would get with onlythree measurements per condition That is why we selected 4 con-ditions to be measured nine times These 4 conditions were chosen

288 m

141 m

245 m

camera

sidelighting

rim lightingstimulus

frontlighting

146 m

Figure 2 Schematic top view of the setup of the photo studio

Figure 3 Example of the triangulation of stimulus A Gaugefigures were only presented on the vertices Subjects never sawthis triangulation during the experiment The triangulation con-sisted of 272 vertices


because they were the most extreme parameter conditions So intotal there were 26 sessions which were presented in randomizedorder Since subjects typically performed one session per day thewhole experiment lasted for several weeks

SubjectsThree naive paid subjects (BZ RH and RS) and 1 nonnaive

subject (PD the f irst author) performed the experiment All hadnormal or corrected-to-normal acuity and good binocular vision asverified with a TNO test (TNO 1972)

ResultsConstructions We constructed surfaces from the

subjectsrsquo settings to get an intuitive feeling for the dataBasically the process came down to fitting the triangu-lation of a smooth surface to the local attitude settingsThus we obtained a depth values map of about 300 pointsof the triangulation Details about such a process can befound elsewhere (see Koenderink et al 1992) These depthvalues maps can be depicted as 3-D surfaces (shapes orreliefs) Representative examples are shown in Figure 4as side views for the nine surfaces of stimulus A for Sub-ject BZ These are ordered in the horizontal direction byincreasing the stereo base in the vertical direction byvarying the lighting condition Clearly these surfaces aresimilar and they resemble a side view of the object thatwas photographed

We noticed some trends in these side views Con-structions with stereo base 14 cm appeared more elongatedin depth whereas with stereo base 0 cm they appearedflatter This can be seen in Figure 4 Another (less obvi-ous) trend was that with a higher stereo base the anglebetween the lower back and the buttocks became morepronounced Also top views were constructed In thesewe noticed that in a few cases the upper part of the bodylooked twisted as compared with the bottom Figure 5shows an example of this for Subject PD with stereobase 14 cm and side and rim lighting It can easily beseen in this figure that the angles between the ldquoshoulderlinesrdquo (lines a and c) and the ldquobuttocks linesrdquo (lines b andd) are different in the two cases shown Another trend we noticed in the top views for stimulus A was that theleft shoulderupper arm looked more curved in the sideand rim lighting conditions Though not as obvious as theformer trend this can also be seen in Figure 5 A localtrend that we noticed in the top views for stimulus B was that the transition between the left and right buttockbecame more conspicuous in the side and rim lightingconditions

Analysis of ResultsScatter plots To further investigate relations between

the different conditions scatter plots of the depth valuesmaps were constructed In order to get a statistically well-balanced set of depth values we only used the first threesettings of the conditions that were measured nine timesfor the construction of scatter plots Later we also usedthese first three settings for the principal componentsanalysis In the scatter plots depth values were plottedfor two conditions either with a different disparity con-dition but everything else (lighting condition stimulusand subject) constant or with a different lighting condi-tion and everything else constant In the remainder ofthis paper we will refer to these scatter plots as the dis-parity and the lighting scatter plots respectively In thedisparity scatter plots those values measured with thelargest of the two stereo bases were always plotted alongthe y-axis Similarly in the lighting scatter plots we plot-ted the depth values obtained in the condition with thelargest angle to the line of sight along the y-axis A lin-

stereo base 0 cm 7 cm 14 cm

lig

htin

g co

nditi

on f

ront

ligh

ting

si

de li

ghti

ng

rim

ligh

ting

Figure 4 Side views of the constructions from the judgmentsof Subject BZ The leftmost column was measured with stereobase 0 cm the middle with 7 cm and the right with 14 cm Theupper middle and bottom rows were measured with lightingcondition rim side and front respectively


ear regression was performed on the data In Figure 6 anexample of the disparity scatter plot is displayed for Sub-ject BZ with rim lighting condition and stereo base14 cm against stereo base 0 cm

With a linear regression the sum of squares of the erroralong the y-axis is minimized Since in our plots no indi-vidual axis was preferred over the other for the slopes weminimized the sum of squares along the line perpendicu-lar to the fitted line As a measure for the goodness of thefit we used the percentage of variance accounted for bythe linear regression These percentages were high Over90 were in the 93ndash99 range with a median ofabout 97 In Table 1 mean slopes and mean goodness-of-fit values averaged over disparity and lighting scatterplots are presented for both stimuli Slopes that differsignificantly from 10 (two-sided students t test p 05df = 8 ts 2306) are marked with an asterisk Sometrends can be seen Mean slopes in the disparity scatterplots were systematically higher than 10 in the lightingscatter plots this was not the case Furthermore there weredifferences between subjects especially Subject RS whodeviated from the others in the slopes and also in thegoodness-of-fit values for disparity which were lower forRS than for the other subjects In the remainder of our

analyses it turned out that results for Subject RS typi-cally differed from the results for the other subjects

A number of different tests were carried out on theslopes and goodness-of-fit values However these testsdid not lead to strong conclusions Therefore we presentonly a rough summary of the results of some tests Onlyfor stimulus A which was measured nine times for fourconditions were data sufficient for further statisticalclaims For Subject RS variation in the goodness-of-fitvalues could be accounted for by random scatter but forthe other 3 subjects this was not the case which indicatesthat for them the cues did indeed have significant effectsHowever the other tests we performed on the scatter plotsdid not clearly reveal what these effects were Thereforein the following section we describe another analysis tech-nique that we used to further investigate the effects of thetwo cues After this analysis it will become evident whythe scatter plots did not reveal these effects clearly

Principal components analysis The initial analysisdid not contradict the idea that the relief depended in acontinuous fashion on the parameter values When onlya single parameter was varied the relief appeared to varymonotonically with the parameter value in complicatedways (eg local effects) differing for the lighting cue

a

b

c

d

Figure 5 Two top views of the constructed surfaces for Sub-ject PD The upper and bottom figures were photographed withlighting condition side and rim respectively Note how in thesetop views the shoulder blades look twisted in one plot with re-spect to the other

0

-40 -30 -20 -10 0 10 20 30 40

depth values at stereo base 0 cm ndashgt

-50

-40

-30

-20

-10

10

20

30

40

50

dept

h va

lues

at s

tere

o ba

se 1

4 cm

ndashgt

pixels

pixels

Figure 6 A scatter plot for Subject BZ On the x-axis we dis-played the condition with stereo base 0 cm and lighting conditionrim on the y-axis we displayed stereo base 14 cm and also light-ing condition rim Also plotted are the lines y = x and the best fit-ting line (slope = 151 goodness of fit = 96) The best fitting lineis displayed as a dotted line


and the stereo cue If this is true one expects that the re-lief can be approximated reasonably well with a linearmodel This is a purely phenomenological analysis in-dependent of any mechanistic model Of course the pa-rameter values themselves (stereo base and angular po-sition of the light source) are quite arbitrary but at leasttheir origins are ecologically significant For a very smallstereo base the disparity cue vanished for almost frontalillumination the shading cue became very weak On thebasis of such very general considerations one would ex-pect a principal component analysis (eg Mardia Kent ampBibby 1977) to yield three major components The firstcomponent should roughly represent the ldquoaverage relief rdquoup to a depth scaling factor and can be said to measure thedegree to which ldquoshape constancyrdquo applies Notice thatwemdashslightly deviating from common usagemdashconsidertwo reliefs that differ only in amplitude to represent theldquosame shaperdquo here This is indeed reasonable in view ofthe inherent ambiguities of the cues One would expectthe second and third components to reflect the influenceof parameter variations Higher order components thenrepresent the influence of nonlinearities and noise Noticethat nonlinearities were certainly expected since a linearmodel is only a first approximation Table 2 shows the vari-ance accounted for by the first three components

As can be seen in Table 2 the first component ac-counted for about 94 of the variance Thus shape con-stancy was the major single component of the responseHowever the residual relief variations were both signif-icant and had systematic regional patterns At least twomore principal components were needed to explain the sys-tematic part of the variance The remainder (roughly 2of the variance) was not significant and failed to show sys-tematic spatial variation but rather appeared noisy withonly some indications of a spatial pattern

Thus the analysis did not contradict our assumptionof a linear model If a linear model applies the effect ofparametric variations of the cues should be revealed inprojections upon the plane spanned by the second andthird principal component (the 2ndash3 plane) In the remain-der of this section we concentrate on this plane This al-lows us to disregard the overwhelming effect of shape

constancy (which turns up in the first dimension) as wellas the scatter in the data (which turns up in the dimensionshigher than the third) In doing this we also discard the in-fluence of nonlinearities which appears to be only slight

One expects the projections on the 2ndash3 plane to havethe general structure suggested in Figure 7 Due to the factthat the actual parameter values are rather arbitrary and(certainly in the case of the illumination) do not allow usto enforce an a priori metric we decided to do a purely or-dinal analysis We simply fit a linear model to the varia-tion in the 2ndash3 plane (again this is pure phenomenologyand implies no specific mechanistic model only continu-ity) and considered the order of the conditions in the majordirections of the 2ndash3 plane

The linear fits were successful except for the patternsfor Subject BZ and RS for stimulus B which were de-generate Notice that the assumption of linearity impliesthat the order of the projections in the 2ndash3 plane reflectsthe order of increase of the parameter values Note thatit is only the order that counts arbitrary reflections and ro-tations or shears are to be disregarded For a random pat-tern the probability of two violations of order is about30 hence we considered cases with two or more viola-tions to be nonsignificant This occurred for Subject RSfor whom no single pattern was significant Thus for this

Table 2The Cumulative Percentage of Variance Accounted

for by the First Three Principal Components

Variance Accounted

Principal for in Percentage

Subject Component Stimulus A Stimulus B

1 940 935BZ 2 964 965

3 981 9781 967 984

PD 2 982 9913 991 9941 940 928

RH 2 961 9653 979 9801 916 949

RS 2 961 9763 978 983

Table 1Mean Slopes and Mean Goodness-of-Fit Valuesof the Fits in the Scatter Plots for Each Subject

for Stimuli A and B Split for the Two Cues

Mean R2 Values and Mean Slopes

Mean Slope Mean Goodness-of-Fit Values ()

Disparity Shading Disparity Shading

Subject Stim A Stim B Stim A Stim B Stim A Stim B Stim A Stim B

BZ 116 096 102 105 97 97 97 96PD 125 113 101 105 99 99 98 99RH 126 121 116 103 97 97 97 96RS 189 106 134 091 95 93 98 97

NotemdashStandard deviation in the mean slopes was about 02 in the goodness-of-fitvalues about 002 The scatter plots were constructed for couples of conditions whichon the x- and y-axes either differed only in lighting condition or in disparity conditionAn asterisk in the slopes indicates a significant deviation from 10


subject all systematic effects were explained by shapeconstancy

For the other subjects only 1 out of 12 patterns (thetotal number) was insignificant (the aforementioned caseof Subject BZ for stimulus B) In most cases there wasone violation and in 3 out of 12 cases there was none

The directions of major variation in the 2ndash3 plane rep-resent the influence on the relief (or the deviation fromshape constancy) due to a pure disparity or illuminationvariation We show these variations in Figure 8 Exceptfor the cases of Subjects BZ and RS with stimulus Bit is apparent that the disparity cue mainly affected theglobal hills and curves in the vertical direction Com-pared with the original shape the bottom and the upperparts were affected so that they looked more elongated(or flattened) The illumination cue mainly caused defor-mations in the horizontal direction which is indicated bythe fact that in the two bottom rows of Figure 8 mainlyvertical lines can be seen The consistency of these effectsover observers is evident even for Subject RS For bothcue variations major changes affected the scapular andpelvic areas for stimulus A a twist in the lumbar area

DISCUSSION

An experiment was performed on the combination ofthe shading and the disparity cue The cues were variedparametrically in a real scene of which photographs weretaken We presented subjects with photos on a computerscreen The subjects performed local attitude settingsfrom which global depth maps were derived These depthmaps were analyzed with a principal components analy-sis It turned out that only three components accountedfor at least 978 of the variance in the data As in sim-ilar experiments (see eg Koenderink et al 1996a) thefirst component accounted for the effect which in liter-ature is known as shape constancy The projections of thedepth maps on the second and third principal compo-nents were calculated It was possible to isolate the ef-

fects for both the disparity and the shading cue variationas linear combinations of the second and third principalcomponents Apparently these isolated effects were(globally) independent of one another We found that theglobal ordinal relations between clusters in the 2ndash3 planewere in the expected order A violation of this would haveindicated interactions In conclusion we found that theeffects of the cues combined linearly The effects of onecue could be added linearly to the effects of the other Thisconclusion is in agreement with the cue combinationmodel of Landy et al (1995) To our knowledge how-ever this has never been confirmed for the shading andthe disparity cue in a realistic setting Furthermore theindividual effects are also in agreement with the litera-ture and in Figure 8 it can be seen that these individualeffects are very similar for each of the subjects We thinkit important to mention here that this strengthens themain conclusion a great deal and is of general interestNext we focus on the individual effects

From our review of the literature we expected globallinear scaling effects for the disparity cue variation Inthe analysis of the scatter plots we found that only for 2subjects did disparity cue variation systematically leadto slopes slightly larger than 10 (Note that in the scat-ter plots on the y-axis depth values were portrayed witha higher stereo base than on the x-axis) This indicatesthat we found only a small global scaling effect Giventhe immediate impression of the stimuli it is perhaps sur-prising that we did not find a larger scaling effect (egsee Figure 1 and compare the impression of monocularlyviewing one photo of a stereo pair to the binocularly fusedimpression)

The plots that show the deformations caused by the dif-ferent cues as depicted in Figure 8 look similar over allsubjects Therefore we could also study a further effectof the disparity cue variation In the two top rows of Fig-ure 8 it can be seen that the constructed objects were af-fected so that the upper and bottom parts appeared moreelongated in depth towards the viewer whereas the mid-dle parts appeared more elongated away from the viewerThus global curvature was affected in the vertical direc-tion but not in the horizontal direction This is true forstimulus A for Subjects BZ RH and PD and less ap-parent for RS and also for stimulus B for Subjects PDand RH We investigated more thoroughly how thisworked It was found that especially the transition betweenthe back and the buttocks became more pronounced withstereo base 14 cm With stereo base 0 cm the back wasslanted backwards but with stereo base 14 cm the backbecame more vertical whereas the upper part of the but-tocks became more horizontal So the third principalcomponent influenced the angle the buttocks made to theback This is not some kind of global scaling effect if itwere the back would have become slanted more backwardinstead of less In Figure 4 this effect can be seen Alsolocally the shoulder blades were affected by the third prin-cipal component In summary we found that subjectsrsquojudgments were globally influenced in the vertical direc-tion in a nonlinear way

0deg 50deg 100degangle with line of sight

0

7

14

ster

eo b

ase

(cm

) 1 2 3

4 5 6

7 8 9

Figure 7 Parameter plot of the conditions On the x-axis thelighting condition is displayed as the angle with the line of sightat which the light source was positioned On the y-axis the stereobase used is displayed in centimeters Numbers 1ndash9 stand for theconditions To reveal the pattern more clearly the conditions areconnected with lines


Figure 8 Deformations of the depth judgments caused by variation of the cues The lightest and the darkest shades of gray depictthe lower and the upper quartiles of the depth respectively the rest is displayed in medium gray Each plot also shows 15 altitudecurves These effects were isolated as a linear combination of the second and third principal components The 4 subjects are depictedin the columns The two top and the two bottom rows are the isolated deformations caused by the disparity and shading cue variationrespectively

subject

bz pd rh rs

ligh

ting

cond

ition

st

ereo

con

ditio

n

stim

ulus

B

stim

ulus

Ast

imul

us B

s

tim

ulus

A

BZ PD RH RS


Next we focus on the effects of the shading cue varia-tion In Figure 8 it can be seen that the plots that describethe deformations caused by the shading cue variationlook very similar for all subjects Mainly vertical lines canbe seen in both stimuli This means that the effects tookplace mainly in the horizontal direction Remember thatthe light source was also moved only in the horizontal di-rection We conclude that there is a connection betweenthe direction of the movement of the light source and thedirection in which effects take place Koenderink et al(1996a) deduced the expectation that shading effects willbe highly correlated with the component of the depth gra-dient in the direction of the light source This expectationwas confirmed in their experiments So if the light sourceis moved in one direction the judged depth gradientshould be altered in the same direction Therefore our re-sults are also in agreement with that expectation

There is another similarity in all deformations forstimulus A (Figure 8)mdashnamely that the upper part of thebody is twisted with respect to the rest of the body Thiscan also be seen in the example depicted in Figure 5Todd et al (1996) also found this twist effect when vary-ing the lighting condition for 1 subject Again we lookedmore thoroughly at all constructions and found that twistsoccurred systematically with the two illumination con-ditions in which the light source was not positioned inthe line of sight In these conditions the right upper partof the object became twisted towards the light source orin other words ldquobrighterrdquo (ie the right shoulder bladeshowed a highlight in these two cases) was judged asldquonearerrdquo This is an effect that was also previously found byKoenderink et al (1996a) However the effect of brighterbeing judged as nearer did not occur for the bottom partof stimulus A which also showed a highlight

One comment should be made about the conclusionstated above The twist did not occur in all circumstancesand in some cases it was barely visible However thefact that it showed up in all four principal componentsanalyses for stimulus A indicated that indeed it was a sys-tematic consequence of the variation of the shading cue

In conclusion in our experiment we found individualeffects for both disparity and shading cue variation whichis in agreement with the literature and we also found thatthe effects of varying both the disparity and the shadingcue combined linearly

REFERENCES

Blake A Zisserman A amp Knowles G (1985) Surface descrip-tions from stereo and shading Image amp Vision Computing 3 183-191

Blakemore C (1969) Binocular depth discrimination and the na-sotemporal division Journal of Physiology 205 471-497

Buckley D amp Frisby J P (1993) Interaction of stereo texture andoutline cues in the shape perception of three-dimensional ridges VisionResearch 33 919-933

Buumllthoff H H amp Mallot H A (1988) Integration of depth mod-ules Stereo and shading Journal of the Optical Society of America A5 1749-1758

Collewijn H amp Erkelens C J (1990) Binocular eye movementsand the perception of depth In E Knowler (Ed) Eye movements andtheir role in visual and cognitive processes (Reviews of OculomotorResearch Vol 4 pp 213-261) Amsterdam Elsevier

Erens R G F Kappers A M L amp Koenderink J J (1993) Per-ception of local shape from shading Perception amp Psychophysics54 145-156

Howard I P amp Rogers B J (1995) Binocular vision and stereopsisOxford Oxford University Press

Johnston E B Cumming B G amp Parker A J (1993) Integrationof depth modules Stereopsis and texture Vision Research 33 813-826

Koenderink J J van Doorn A J Christou C amp Lappin J S(1996a) Perturbation study of shading in pictures Perception 251009-1026

Koenderink J J van Doorn A J Christou C amp Lappin J S(1996b) Shape constancy in pictorial relief Perception 25 155-164

Koenderink J J van Doorn A J amp Kappers A M L (1992) Sur-face perception in pictures Perception amp Psychophysics 52 487-496

Koenderink J J van Doorn A J amp Kappers A M L (1995)Depth relief Perception 24 115-126

Landy M S Maloney L T Johnston E B amp Young M (1995)Measurement and modelling of depth cue combination In defense ofweak fusion Vision Research 35 389-412

Landy M S Maloney L T amp Young M (1991) Psychophysicalestimation of the human depth combination rule In P S Schenker(Ed) Sensor fusion III 3-D perception and recognition (Proceedingsof the SPIE 1383 pp 247-254)

Mardia K V Kent J T amp Bibby J M (1977) Multivariate analy-sis London Academic Press

Marr D (1982) Vision San Francisco FreemanNorman J F Todd J T amp Phillips F (1995) The perception of

surface orientation from multiple sources of optical information Per-ception amp Psychophysics 57 629-636

Ramachandran V S (1988 August) Perceiving shape from shadingScientific American 331 133-166

Stevens K A amp Brookes A (1988) Integrating stereopsis withmonocular interpretations of planar surfaces Vision Research 28371-386

TNO (1972) TNO test for stereoscopic vision (8th ed) UtrechtLameacuteris Instrumenten BV Institute for Perception

Todd J T Koenderink J J van Doorn A J amp Kappers A M L(1996) Effects of changing viewing conditions on the perceivedstructure of smoothly curved surfaces Journal of Experimental Psy-chology 22 695-706

Todd J T Norman J F Koenderink J J amp Kappers A M L(1997) Effects of texture illumination and surface reflectance onstereoscopic shape perception Perception 26 807-822

Young M J Landy M S amp Maloney L T (1993) A perturbationanalysis of depth perception from combinations of texture and mo-tion cues Vision Research 33 2685-2696

NOTE

1 This is the square of the length of the longest semi axis of the co-variance ellipse divided by the sum of the squares of the lengths of bothsemi axes of the covariance ellipse (Mardia et al 1977)

(Manuscript received December 31 1998revision accepted for publication October 25 2000)


parity By texture we mean variation of the surface albedorather than three-dimensional (3-D) surface roughnessLike dihedral edges texture is well localized and there-fore leads to correct disparity Radiance variation due toattached and cast shadow boundaries also creates possi-bilities for useful binocular matches The specular com-ponent leads to matches whose disparities do not revealthe surface in a simple way because the spatial structureof this component depends on the vantage point

In the present experiment we varied the disparity cueparametrically by varying the stereo base from zero toabout twice the natural stereo base in pairs of stereo pho-tographs When the stereo base is varied three parameterswill be altered the absolute disparity and with it the rel-ative disparity (see Blakemore 1969 for the distinctionbetween the two) and the vergence It is important to notethat a change of any of these parameters will globally af-fect the percept of a scene Collewijn and Erkelens (1990)argued that any role played by vergence-related signals orabsolute disparity in the estimation of distance can onlybe weak Under full cue conditions other cues dominateHowever binocular stereopsis is sensitive to relative dis-parities and relative disparity is an effective cue to rela-tive depth differences From geometrical considerationsit can be concluded that when the stereo base is alteredrelative disparity will signal a differently scaled dispar-ity f ield Therefore the same object viewed with ahigher stereo base might be interpreted as being moreelongated in depth (see also Howard amp Rogers 1995 fora theoretical discussion on stereopsis) On those groundswe expect global scaling effects to occur when the stereobase is varied This expectation is based only on the as-sumption that the visual system presumes a constantstereo base which indeed seems to be the case (Howardamp Rogers 1995)

In the past such scaling effects have indeed been foundin binocular viewing experiments in which the stereobase was varied (Koenderink van Doorn amp Kappers1995) The natural stereo base of the subjects and stereobase zero were used It was found that in the zero dispar-ity condition shapes were globally judged to be flattenedHowever in these experiments unlike in the present ex-periment subjects looked at a real scene instead of pho-tographs Note that in the stereo base zero condition rel-ative disparity signals a disparity field with zero depthso if the observer had based shape judgments on dispar-ity alone shapes would have been judged to be totally flatwhich was not the case

We mentioned that the disparity and the shading cueprovide complementary shape information and dependon different prior assumptions We pointed out that theshading cue is not of a global nature and it also has anambiguous nature In contrast the disparity cue is of aglobal nature When a scene is illuminated by for in-stance two light sources local parts of the scene may beilluminated by only one source However there is no waythat we can for instance have a real scene with one stereobase on one part and another one on another part Anotherdifference is the fact that shading variations depend

mainly on the surface curvature but disparity dependsonly on the distance For instance if we look at a uniformlyilluminated flat slanted surface in isolation shading pro-vides us with no information whatsoever about this slant(or the depth) In that case disparity is still available

We mentioned that shading can help to provide dis-parity information by means of for instance attachedshadow boundaries On the other hand disparity informa-tion can be used to disambiguate shading (eg Blake et al1985) So the two cues can be complementary There-fore it will be very interesting to see whether and if sohow these two cues are combined and whether effects ofdisparity variation depend on the lighting of the scene

Buumllthoff and Mallot (1988) addressed the issue of cueinteraction between disparity and shading on simple ob-jects (ellipsoids) with computer-generated shading Theyfound that the amount of elongation in depth of judgedsurfaces was an accumulation of the contribution of thetwo cues They also found that despite the smooth natureof radiance variation due to the Lambertian componentthis could still be used as a feature for useful binocularstereopsis A major difference between our experiment andthe experiment of Buumllthoff and Mallot is that we use amuch more complex stimulus

Landy Maloney Johnston and Young (1995) reviewedthe literature on cue combination and interaction and pre-sented a formal model of depth cue combination that isconsistent with many experimental results The main pur-pose of this model is not to describe how the visual sys-tem works but rather to guide cue combination experi-ments The model assumes a depth system that is dividedinto different modules for different depth cues In eachmodule a depth map is ldquocomputedrdquo and also the relia-bility of this depth map is estimated This reliability canvary between positions in a scene The various depth esti-mates are combined into a single depth map by a weightedlinear average where the weights take into account theestimated reliability and also possible discrepancies be-tween depth maps By also accounting for these discrep-ancies in the final depth estimate Landy et al (1995)stressed that their model becomes robust which meansthat if the deviation of one cue as compared with theother cues increases from zero it should affect the finaldepth estimate linearly However when the deviations be-come larger and increase beyond the range present in nor-mal scenes the influence of this one discrepant cue is as-sumed to have less and less effect on the percept In themost extreme case a cue can even be vetoed in this modelif the information from this cue is too different from theinformation supplied by other cues

Next we bring up some issues of methodology A com-mon way to investigate the efficacy of a single cue is totry and isolate it from all the others In most cases how-ever one obtains ambiguous results For instance in thecase of shading subjects fail to obtain a clear impressionof depth when other cues are totally absent (eg ErensKappers amp Koenderink 1993) An alternative is to varya single cue parametrically while all others are kept con-stant Such a perturbation analysis has been used success-






METHOD




a b a

c d c

e f

g h g


A B

C D C

E F

G H G










288 m

141 m

245 m

camera

sidelighting


frontlighting

146 m













lig

htin

g co

nditi

on f

ront

ligh

ting

si

de li

ghti

ng

rim

ligh

ting








a

b

c

d


0

-40 -30 -20 -10 0 10 20 30 40


-50

-40

-30

-20

-10

10

20

30

40

50

dept

h va

lues

at s

tere

o ba

se 1

4 cm

ndashgt

pixels

pixels











Variance Accounted



1 940 935BZ 2 964 965

3 981 9781 967 984

PD 2 982 9913 991 9941 940 928

RH 2 961 9653 979 9801 916 949

RS 2 961 9763 978 983







BZ 116 096 102 105 97 97 97 96PD 125 113 101 105 99 99 98 99RH 126 121 116 103 97 97 97 96RS 189 106 134 091 95 93 98 97






DISCUSSION






0

7

14

ster

eo b

ase

(cm

) 1 2 3

4 5 6

7 8 9




subject

bz pd rh rs

ligh

ting

cond

ition

st

ereo

con

ditio

n

stim

ulus

B

stim

ulus

Ast

imul

us B

s

tim

ulus

A

BZ PD RH RS






REFERENCES
























NOTE








METHOD




a b a

c d c

e f

g h g


A B

C D C

E F

G H G










288 m

141 m

245 m

camera

sidelighting


frontlighting

146 m













lig

htin

g co

nditi

on f

ront

ligh

ting

si

de li

ghti

ng

rim

ligh

ting








a

b

c

d


0

-40 -30 -20 -10 0 10 20 30 40


-50

-40

-30

-20

-10

10

20

30

40

50

dept

h va

lues

at s

tere

o ba

se 1

4 cm

ndashgt

pixels

pixels











Variance Accounted



1 940 935BZ 2 964 965

3 981 9781 967 984

PD 2 982 9913 991 9941 940 928

RH 2 961 9653 979 9801 916 949

RS 2 961 9763 978 983







BZ 116 096 102 105 97 97 97 96PD 125 113 101 105 99 99 98 99RH 126 121 116 103 97 97 97 96RS 189 106 134 091 95 93 98 97






DISCUSSION






0

7

14

ster

eo b

ase

(cm

) 1 2 3

4 5 6

7 8 9




subject

bz pd rh rs

ligh

ting

cond

ition

st

ereo

con

ditio

n

stim

ulus

B

stim

ulus

Ast

imul

us B

s

tim

ulus

A

BZ PD RH RS






REFERENCES
























NOTE












288 m

141 m

245 m

camera

sidelighting


frontlighting

146 m













lig

htin

g co

nditi

on f

ront

ligh

ting

si

de li

ghti

ng

rim

ligh

ting








a

b

c

d


0

-40 -30 -20 -10 0 10 20 30 40


-50

-40

-30

-20

-10

10

20

30

40

50

dept

h va

lues

at s

tere

o ba

se 1

4 cm

ndashgt

pixels

pixels











Variance Accounted



1 940 935BZ 2 964 965

3 981 9781 967 984

PD 2 982 9913 991 9941 940 928

RH 2 961 9653 979 9801 916 949

RS 2 961 9763 978 983







BZ 116 096 102 105 97 97 97 96PD 125 113 101 105 99 99 98 99RH 126 121 116 103 97 97 97 96RS 189 106 134 091 95 93 98 97






DISCUSSION






0

7

14

ster

eo b

ase

(cm

) 1 2 3

4 5 6

7 8 9




subject

bz pd rh rs

ligh

ting

cond

ition

st

ereo

con

ditio

n

stim

ulus

B

stim

ulus

Ast

imul

us B

s

tim

ulus

A

BZ PD RH RS






REFERENCES
























NOTE













lig

htin

g co

nditi

on f

ront

ligh

ting

si

de li

ghti

ng

rim

ligh

ting








a

b

c

d


0

-40 -30 -20 -10 0 10 20 30 40


-50

-40

-30

-20

-10

10

20

30

40

50

dept

h va

lues

at s

tere

o ba

se 1

4 cm

ndashgt

pixels

pixels











Variance Accounted



1 940 935BZ 2 964 965

3 981 9781 967 984

PD 2 982 9913 991 9941 940 928

RH 2 961 9653 979 9801 916 949

RS 2 961 9763 978 983







BZ 116 096 102 105 97 97 97 96PD 125 113 101 105 99 99 98 99RH 126 121 116 103 97 97 97 96RS 189 106 134 091 95 93 98 97






DISCUSSION






0

7

14

ster

eo b

ase

(cm

) 1 2 3

4 5 6

7 8 9




subject

bz pd rh rs

ligh

ting

cond

ition

st

ereo

con

ditio

n

stim

ulus

B

stim

ulus

Ast

imul

us B

s

tim

ulus

A

BZ PD RH RS






REFERENCES
























NOTE









a

b

c

d


0

-40 -30 -20 -10 0 10 20 30 40


-50

-40

-30

-20

-10

10

20

30

40

50

dept

h va

lues

at s

tere

o ba

se 1

4 cm

ndashgt

pixels

pixels











Variance Accounted



1 940 935BZ 2 964 965

3 981 9781 967 984

PD 2 982 9913 991 9941 940 928

RH 2 961 9653 979 9801 916 949

RS 2 961 9763 978 983







BZ 116 096 102 105 97 97 97 96PD 125 113 101 105 99 99 98 99RH 126 121 116 103 97 97 97 96RS 189 106 134 091 95 93 98 97






DISCUSSION






0

7

14

ster

eo b

ase

(cm

) 1 2 3

4 5 6

7 8 9




subject

bz pd rh rs

ligh

ting

cond

ition

st

ereo

con

ditio

n

stim

ulus

B

stim

ulus

Ast

imul

us B

s

tim

ulus

A

BZ PD RH RS






REFERENCES
























NOTE












Variance Accounted



1 940 935BZ 2 964 965

3 981 9781 967 984

PD 2 982 9913 991 9941 940 928

RH 2 961 9653 979 9801 916 949

RS 2 961 9763 978 983







BZ 116 096 102 105 97 97 97 96PD 125 113 101 105 99 99 98 99RH 126 121 116 103 97 97 97 96RS 189 106 134 091 95 93 98 97






DISCUSSION






0

7

14

ster

eo b

ase

(cm

) 1 2 3

4 5 6

7 8 9




subject

bz pd rh rs

ligh

ting

cond

ition

st

ereo

con

ditio

n

stim

ulus

B

stim

ulus

Ast

imul

us B

s

tim

ulus

A

BZ PD RH RS






REFERENCES
























NOTE







DISCUSSION






0

7

14

ster

eo b

ase

(cm

) 1 2 3

4 5 6

7 8 9




subject

bz pd rh rs

ligh

ting

cond

ition

st

ereo

con

ditio

n

stim

ulus

B

stim

ulus

Ast

imul

us B

s

tim

ulus

A

BZ PD RH RS






REFERENCES
























NOTE





subject

bz pd rh rs

ligh

ting

cond

ition

st

ereo

con

ditio

n

stim

ulus

B

stim

ulus

Ast

imul

us B

s

tim

ulus

A

BZ PD RH RS






REFERENCES
























NOTE








REFERENCES
























NOTE



Documents

Doorschot (2001) The combined influence of binocular