The perception of scale-dependent and scale-independent ...€¦ · Perception, 1997, volume 26, pages 147-166 The perception of scale-dependent and scale-independent surface structure

Perception, 1997, volume 26, pages 147-166

The perception of scale-dependent and scale-independent surface structure from binocular disparity, texture, and shading

James S Tittle, J Farley Norman % Victor J Perotti, Flip Phillips Department of Psychology, Ohio State University, 142 Townshend Hall, 1885 Neil Avenue Mall, Columbus, OH 43210-1222, USA; e-mail: [email protected]; If Department of Psychology, Western Kentucky University, Bowling Green, KY 42101, USA Received 3 July 1996, in revised form 14 December 1996

Abstract. The integration of binocular disparity, shading, and texture was measured for two different aspects of three-dimensional structure: (1) shape index, which is a measure of scale-independent structure, and (2) curvedness, which is a measure of scale-dependent structure. Binocular disparity was found to contribute significantly more to judged shape index than it does to judged curvedness, and shading and texture were both found to contribute more to judged curvedness than to judged shape index. These results demonstrate that different cues do not contribute equally to different aspects of perceived surface structure. This finding suggests that, for the case of linear integration, multiple cues to three-dimensional structure do not combine on the basis of a single type of representation shared by all the 'shape-from-X' processes in the visual system.

1 Introduction Many distinct sources of optical information specify the three-dimensional (3-D) shape of objects and surfaces in our environment. Consequently, researchers over the past decade have begun examining how the visual system combines different types of cues to arrive at a perception of 3-D structure. These studies have included such combinations as motion, occlusion, and relative size (Bruno and Cutting 1988), binocular disparity and motion (Rogers and Collett 1989; Tittle and Braunstein 1993; E B Johnston et al 1994; Tittle et al 1995), disparity and shading (Bulthoff and Mallott 1988; Norman et al 1995), and disparity and texture (Stevens et al 1991; E B Johnston et al 1993; Buckley and Frisby 1993). As this partial list shows, many different combinations of cues have been studied. However, one important aspect of this integration of multiple cues has not received much attention: what information about 3-D shape is being integrated. Most studies have been based on the working assumption that local depth estimates are being combined from the various cues [see Cutting and Vishton (1995), Norman et al (1995), and Stevens (1995) for notable exceptions], but depth is only one of many possible representations of 3-D structure.

The problem is that we do not have uniform agreement in the literature about what it means to perceive the 3-D shape of an object or surface (for example, 'shape' could refer to depth, orientation, curvature, or some combination of properties). Therefore, before we can fully understand the integration of multiple cues we need to know what properties of 3-D structure are being integrated. Distinguishing between these possibilities is particularly important when considering the different types of optical information that specify 3-D shape. For example, theoretical analyses demonstrate that while relative depth can in principle be recovered directly from binocular disparities (Mayhew and Longuet-Higgins 1982) or motion (Ullman 1979), it is less likely that this information is recovered directly from shading (Koenderink and van Doom 1980; Horn and Brooks 1989) or texture (Gibson 1950; Witkin 1981).

Recent empirical studies of perceived shape from disparity, texture, and shading also support the idea that these sources do not directly provide relative depth. McKee et al

mailto:[email protected]

148 J S Tittle, J F Norman, V J Perotti, F Phillips

(1990) have demonstrated that depth discriminations from binocular disparity are relatively imprecise as stimuli move away from the horopter. Furthermore, Norman and Todd (1996) have reported that the variability of judged depth is greater than that of orientation, while A Johnston and Passmore (1994b) found that the variability of judged orientation is in turn greater than that of curvature. The finding that observers' perceived local 3-D shape becomes less variable as one measures higher-order surface structure suggests that their responses were not based on a depth map. Thus, for disparity, texture, and shading it seems quite possible that the visual system recovers relative slant or curvature without first computing local depth. Because of this uncertainty about the information specified by 3-D cues, a full understanding of local-shape perception must entail not only a description of those properties represented by the visual system, but also the degree to which each is specified by the various optical sources of information. The purpose of the experiments presented here was to begin this process by examining the integration of two aspects of 3-D shape from binocular disparity, texture, and shading: scale-independent and scale-dependent surface structure.

Some specific examples of local 3-D surface properties can be seen by considering a two-parameter family of quadric surfaces of the form

z(x,y) = \{KXX2 + K2y2), (1)

where z is depth along the line of sight, x and y represent coordinates in the image plane, and K1 and K2 are the parameters whose values determine the specific quadric surface being represented. Such surfaces have traditionally been described by using quantitative measures of each local region on the surface. The three local measures most commonly mentioned have been (i) depth, (ii) orientation, and (iii) curvature. The set of points (x, y, z) satisfying equation (1) provide the most basic description of a surface, and are commonly referred to as a depth map. Although such a representation may be sufficient to perform many visual tasks its primary drawback is that it lacks viewpoint independence (cf Mamassian et al 1996). Another problem with the depth map stems from results showing extensive anisotropics in the perception of orientation (Gillam et al 1988) and curvature (Rogers and Graham 1983). Such anisotropics are not observed in subjects' depth judgments and thus are not adequately explained by a depth map.

By considering the first spatial derivative of the depth map one obtains a representation based not on local depth but rather on local surface orientation. The surface normal (n) describes how quickly depth changes along any particular direction on the surface. Although the surface normal provides a translation-invariant representation of 3-D structure, the value of the normal is not invariant under arbitrary rotation. Thus, it is still viewpoint or coordinate-system dependent.

By measuring the rate at which orientation changes in a particular direction along a smoothly curved surface (ie its second-order surface structure) we can obtain a local surface property that is not coordinate-system dependent (Kreyszig 1959). Specifically, for any point on a smooth nonplanar surface there will always be one direction in which the curvature reaches a maximum (fcmax) and an orthogonal direction in which the curvature reaches a minimum (?cmin). For the surface described by equation (1), the curvatures along the x-axis and jy-axis are given by

Binocular disparity, texture, and shading 149

At the origin these are the directions of the principal curvatures, and from equations (2) and (3) it can be seen that their magnitudes are simply the parameters /c, and K2.

The principal curvatures Kmax and fcmin provide a coordinate-system-independent description of the second-order structure of a quadric surface, but they still have certain properties that seem to be inconsistent with our perceptual experience. For example, on the basis of our perceptual ability to simultaneously perceive both scale-independent (eg spherical vs cylindrical) and scale-dependent (eg magnitude-of-curvature) aspects of shape, it would be desirable to have shape descriptors that independently represent these attributes.

Koenderink (1990) has recently proposed two such measures of local 3-D structure: shape index and curvedness. These measures are based on a polar coordinate transformation of Kmax and Kmin space to separate out their scale-independent (shape-index) and scale-dependent (curvedness) components. The shape index and curvedness are local measures of surface structure which would normally vary widely across natural surfaces. However, because these measures are relatively constant across the class of quadric surfaces we will consider throughout the rest of the paper that the shape index and curvedness of a surface at the origin will be used as global measures of 3-D form. The shape index (S) provides a one-dimensional description of shape independent of the magnitude of curvature:

S = - - arctan Kmax + *min . (4) ft ^max ^min

Variations in the scale-dependent surface structure are referred to as the local curvedness (C):

(2 2 \ 1/2

2 ) • (5)

Several recent studies have examined the perception of local scale-independent surface structure. The evidence from studies involving binocular stereopsis (de Vries et al 1993), motion (van Damme et al 1994), and shading (Mamassian et al 1996) is consistent with the idea that human observers can accurately judge shape index independently of variations in curvedness. However, it is worth noting that Erens et al (1993) have reported that human observers could not discriminate local shape index from Lambertian-shaded quadric surfaces. Despite the ambiguity regarding shape from shading, these results suggest to us that scale-independent (eg shape-index) and scale-dependent (eg curvedness) aspects of local surface structure play an important role in the perception of 3-D form, and in the experiments to follow we will seek to determine their influence on the integration of multiple cues.

We have argued that when studying integration of multiple cues an important issue is the extent to which different optical sources contribute to different aspects of 3-D structure. Some recent evidence on shape perception from binocular stereopsis and motion indicates that there are important differences in the aspects of surface structure specified by these sources. Perotti et al (1995) found differences between structure from motion and binocular stereopsis when they performed stereo matches of motion-defined surfaces simulating different shape-index and curvedness values. They observed a very high correlation between simulated and matched shape index, but a much lower correlation between simulated and matched curvedness. In addition, Tittle and Perotti (1997) measured the relative contribution of motion and binocular disparity in a cue-conflict paradigm and found a similar pattern of results: (i) motion contributed more to judged shape index than to curvedness, and (ii) binocular disparity contributed more to judged curvedness than to shape index.


These results indicate that the relative contribution of motion and stereo depends upon whether one is measuring scale-independent or scale-dependent aspects of surface structure. Such findings are not only interesting for the specific case of perceived shape from motion and stereo, but also have important implications for general models of 3-D-shape perception from multiple cues. Specifically, they suggest that the visual system might have multiple parallel cue integrations which coincide with different surface representations. Thus, it is important to determine if other sources of 3-D shape exhibit the disassociation between scale-independent and scale-dependent 3-D structure that we observed for binocular stereopsis and structure from motion. To determine if this finding is indicative of other static sources of 3-D shape we measured observers' perceived shape index and curvedness for displays with conflicting binocular disparity, shading, and texture.

2 General method 2.1 Subjects The first three authors (JST, JFN, and VJP) and one additional experienced psychophysical observer (JTT) participated in all three experiments.

2.2 Apparatus and stimuli Our stimuli were generated and presented on a Silicon Graphics Crimson VGXT workstation. The physical resolution of this system was 1280 x 1024 pixels, but with antialiasing we were able to increase the effective resolution by a factor of 10. The left and right views of our binocular stereograms were presented with an LCD shuttered system (Stereographies—Crystal eyes) that was synchronized to the monitor refresh rate (120 Hz). The results of direct photometric measurement and a psychophysical flicker-photometry procedure revealed that the leakage of light from the red, green, and blue channels of the display monitor into the inappropriate image in a stereo pair was 1.4%, 3.8%, and 4.0%, respectively. At our maximum luminance (20.7 cd m"2

as measured through the LCD stereoglasses) this led to a leakage of approximately 0.05 cd m - 2 for the red, 0.53 cd m~2 for the green, and 0.08 cd m - 2 for the blue channel. Our observers viewed the stimuli in a darkened room seated 114.6 cm away from the monitor with their head stabilized in a chin rest.

All stimuli used in our experiments were quadric surfaces satisfying equation (1). These surfaces varied only in the specific values of the parameters K1 and K2, and on each trial these values were chosen by solving equations (5) and (6) for a particular shape index and curvedness. For the set of experiments described here the shape index varied in five steps between 0.0 and 1.0, and the curvedness in five steps between 0.75 and 1.75 cm"1. The twenty-five shapes generated from the combination of five shape-index and five curvedness values can be seen in figure 1.

The quadic surfaces were covered with approximately 144 circular texture elements arranged in a grid pattern on the surface. However, all of these texture elements were not visible because we presented a circular cutout of the surface with diameter 8 deg visual angle to eliminate any occluding contour shape cues. The surface background color was generated by using only the blue channel of the monitor (dominant wavelength equal to 467 nm) and the circular texture elements were generated by using only the red channel (dominant wavelength equal to 623 nm). The texture elements subtended a visual angle of 0.64 deg2 when they were projected as full circles in the image. Of course, this only happened when the surface normal at the center of the element was coincident with the line of sight. We will describe below how the projected images of the texture elements were computed for any surface-normal orientation.

The texture cue in our displays was created on the basis of the assumption that elements were lying in the tangent plane defined at the center of the circular element.


0.0 0.25 0.5 0.75 1.0 Shape index, S

Figure 1. The specific shape-index and curvedness values used in our experiments.

This approximation leads to textures in the image that will be elliptical in shape. The eccentricity and orientat ion of these projected ellipses is a function of only the surface normal (see appendix), and the projected size of the element is a function of its distance from the observer. However, because the depth range of our surfaces was small (approximately 8 cm) relative to the viewing distance (114.6 cm) almost all of the texture variat ion is due to changes in the normal as one moves across the surface. This should not pose any part icular problem, because studies of perceived shape from texture have indicated that a lmost all of the perception of curved surfaces can be accounted for by the component related to the orientation of the surface normal (Cutting and Millard 1984; Todd and Akers t rom 1987; E B Johnston et al 1993).

The second cue we used to simulate our quadric surfaces was shading with specular highlights. Specifically, the intensity of each vertex in the polygon mesh which defined our surface was specified by the Phong illumination model (Foley et al 1990). G o u r a u d shading (vertex intensity interpolation) was then used to define the intensity at each location in the image.

Earlier we described how the texture deformation in the image was based on the assumption tha t the elements were lying in the tangent plane of the surface. However, in the final display they were mapped back onto the binocular-disparity-defined surface (see appendix). This enabled us to generate textures that simulated one quadric surface and binocular disparities that simulated another. Binocular disparities were created by presenting separate views to the left and right eyes on the basis of an interocular separation of 6.25 cm and a viewing distance of 114.6 cm. Examples of conflicting binocular disparity, texture, and shading defined quadric surfaces can be seen in figures 2 and 3.

These cue-conflict quadric surfaces were created by independently choosing the appropriate z-coordinates for the stereo-defined surface, and then independently choosing the appropriate normals for bo th the texture-defmed and the shading-defined surface. To simultaneously depict these three surfaces we then computed the binocular disparity of texture elements on the basis of (x, y , zdispari ty), the texture deformation on the basis of using wteXture m the texture equations described in the appendix, and nsh&ding to compute local image intensity in the illumination model.


il^iiHill l l i^pi|| l l^(ii^

IHII^ri|tanli^ri|teiK:HH' jsSsSSl^siifttiKSSiiJSss^ mmmffffiifiW

iIlllliBilpBijpii':,i

;> : i :S* | :v :vX^

Figure 2. Sample cue-conflict stereograms depicting two quadric surfaces with the same shape index (0.0) but different curvedness values. The stereogram is designed for cross fusion.

Figure 3. Sample cue-conflict stereograms depicting two quadric surfaces with the same curvedness but different shape-index values. The stereogram is designed for cross fusion.


2.3 Procedure The paradigm we used required the observers to match the perceived 3-D structure of the display with stereo - texture - shading conflict with an adjustable display with consistent stereo, texture, and shading. However, we wanted to be sure that observers could not simply match two-dimensional (2-D) features between the two displays. Thus, our displays with consistent stereo, texture, and shading differed from the conflict displays in a number of important ways. First, we replaced the circle textures with a blue - red Julesz random-dot texture. This was done so that observers would not make their judgments by matching the local shape of texture elements between the conflict and consistent displays. Second, we randomly varied the position of the light source and removed the specular component for the consistent displays. This precaution was taken so that observers would not be able to respond by simply matching the 2-D intensity pattern across the two displays. Last, to help ensure that observers were not matching the relative disparities around the edge of the display, we replaced the smooth circular cutout of the surface with a randomly computed jagged edge. The results of these changes can be seen in figure 4. In all of our experiments the observers' task was to adjust the principal curvatures (K{ and K2) of the display with consistent stereo, texture, and shading until it appeared to match the perceived 3-D structure of the conflict display.

Figure 4. Sample stereograms depicting the adjustable response surfaces set to shape-index values of 0.5 (top) and 1.0 (bottom).

3 Experiment 1: Judged shape index from binocular disparity, texture, and shading In the first experiment we presented observers with quadric surface displays in which stereo, texture, and shading all simulated the same curvedness (one of the five values from figure 2) but different shape-index values. The perception of shape index has been studied for stereo (de Vries et al 1993) and shading (Erens et al 1993) in isolation, and the results from these studies suggest that shading will contribute little to judged


shape index. There have been no previous studies of shape from texture that have specifically looked at scale-independent surface structure, but results from Todd and Akerstrom (1987) and E B Johnston et al (1994) would seem to indicate that texture will at least provide a significant contribution to judged curvedness. The purpose of this first experiment was to measure the relative contribution of stereo, texture, and shading to judged shape index and then compare this with the judgments of curvedness that were obtained in experiment 2.

3.1 Procedure Each of our four observers judged the perceived 3-D structure of 200 quadric surface patches. These stimuli were chosen from random combinations of the twenty-five canonical shapes shown in figure 2. The three sources had the same curvedness on each trial, between 0.75 cm-1 and 1.75 cm"1, but the shape index simulated by each source was randomly chosen from the five values between 0.0 and 1.0. Thus, at the beginning of each trial a curvedness value (corresponding to one of the rows in figure 1) was randomly chosen and then each of the three sources was independently assigned a shape-index value corresponding to one of the columns in that row. Example stimuli in which binocular disparity simulates one shape index and texture and shading simulate another can be seen in figure 3.

At the beginning of each trial the conflict stimulus would appear, and the subject's task was to judge the curvature along the two principal directions. This judgment was performed by matching the conflict stimulus to a surface display with consistent stereo, texture, and shading such as those portrayed in figure 4. Observers were allowed to toggle back and forth between the two displays and adjust principal curvatures (fCj and K2) of the matching stimulus until a satisfactory match was achieved.

3.2 Results and discussion \ In this experiment observers judged the principal curvatures of conflict displays in which stereo, texture, and shading all simulated the same curvedness, but different shape-index values. From the adjusted KX and K2 values recorded on each trial we computed the observers' shape-index judgment by using equation (5). To measure the relative contribution of each source to the observers' final judgment we then performed a multiple linear regression analysis on our observers' judged shape index as a function of the simulated stereo, texture, and shading shape index. Specifically we fit each observer's data to the simple linear model:

fudged = ^ s t ^ s t + WXSX + ^ s h S s h , ( 6 )

where Sjudged equals the observers' judged shape index computed from the adjusted KX

and K2, and Ssi, St, and Ssh are the shape-index values simulated by binocular disparity, texture, and shading, respectively. Last, to provide a measure of the contribution of each source to the final judgment, we obtained estimates of Wst, Wt, and Wsh, which are the weights assigned to stereo, texture, and shading, respectively.

The results of the multiple linear regression analysis can be seen in table 1. The multiple-i?2 values in the left column indicate that the linear model provided an excellent fit of our observers' shape-index judgments. Such a result supports theories of cue integration based on a weighted linear combination (Landy et al 1995). However, we have previously found evidence for significant nonlinearities in cue integration for near-threshold tasks (Tittle et al 1997).

The data in table 1 also show that, while all three sources contribute to judged shape index, the weight assigned to the binocular-disparity information is much larger than that obtained for the other two cues. This result is consistent with previous studies involving the integration of binocular disparity with either shading (Bulthoff and Mallott 1988) or texture (E B Johnston et al 1993). Although it does appear that


Table 1. KX and

Subject

Results of the multiple regression analysis for the shape-index values computed from the K2 judgments in experiment 1.

Multiple R2

JFN 0.90 JST 0.95 JTT 0.98 VJP 0.95

Note: standard errors

w« wx

0.708 (0.031) 0.138 (0.031) 0.809 (0.020) 0.090 (0.021) 0.902 (0.020) 0.040 (0.011) 0.782 (0.022) 0.089 (0.021)

appear in parentheses.

^ s h

0.092 (0.032) 0.039 (0.020) 0.032 (0.011) 0.064 (0.021)

stereo dominated texture and shading in specifying shape index our primary interest is to compare the relative contribution of each source to shape index and curvedness. Thus, in experiment 2 we measured perceived 3-D structure for stereo - texture - shading displays in which the three sources simulated different curvedness values.

4 Experiment 2: Judged curvedness from binocular disparity, texture, and shading 4.1 Procedure The task in this experiment was identical to that used in experiment 1: observers judged the 3-D structure of 200 quadric surfaces by adjusting the principal curvatures of a cue-consistent matching display. The only difference between this and the previous experiment was the set of cue-conflict stimuli viewed by our observers. Unlike in the first experiment, we now used displays in which disparity, texture, and shading simulated the same shape index but conflicted with regard to curvedness. Thus, on each trial a shape index was chosen from one of the five columns in figure 1 and that value was assigned to all three cues. Next we randomly assigned a curvedness to disparity, texture, and shading from amongst the values shown in the five rows in figure 1. As in the first experiment, observers matched the perceived 3-D structure of the conflict stimulus by adjusting the principal curvatures of a display with consistent disparity, texture, and shading.

4.2 Results and discussion Because the disparity, texture, and shading information differed only in terms of curvedness for this experiment, we computed judged curvedness from the KX and K2

adjustments of each observer by using equation (5). The weights associated with each source of optical information were then estimated by using a multiple regression analysis to fit observers' responses to the curvedness version of the linear model described by equation (6). From table 2 we can again see that this linear model provides a very good fit of the data.

Table 2. Results of the multiple regression analysis for the curvedness values computed from the K] and K2 judgments in experiment 2.

Subject Multiple Wst Wt Wsh

JFN JST JTT VJP

0.98 0.97 0.98 0.95

0.338 (0.017) 0.372 (0.023) 0.465 (0.019) 0.331 (0.016)

0.199 (0.017) 0.213 (0.023) 0.109 (0.018) 0.197 (0.017)

0.114 (0.017) 0.104 (0.022) 0.064 (0.018) 0.108 (0.017)

Note: standard errors appear in parentheses.


As in experiment 1, the weights assigned to stereo are uniformly larger than those assigned to the other two sources. However, in this case the difference between disparity, texture, and shading seems to be less than that observed for perceived shape index. However, the weights in tables 1 and 2 are in different units, so to make direct comparisons across experiments they must be normalized. Therefore, in figure 5 we have plotted the weights estimated from experiments 1 and 2 based on a z-score transformation of the shape-index and curvedness values. For all observers a clear pattern emerges from the standardized regression coefficients shown in figure 5: binocular stereopsis contributes significantly more to judged shape index than to judged curvedness, whereas both texture and shading contribute more to judged curvedness than to shape index.

0.90

0.75

I | 0.60

a 0.45 o

g, 0.30

0.15

0

JFN

Iss

T m j i '-it!

H shape index

curvedness

0.90 J

0.75 H

0.60

0.45

0.30

0.15

VJP

^

* j * •!"

T

T

1 ̂ tfz T

nfci _ ^

i > ' r

H.ii.mili'i

Stereo Texture Shading

JTT

T . JES3SL

p T


Figure 5. Bar graphs showing the standardized regression coefficients of binocular disparity (stereo), texture, and shading from experiment 1 (judged shape index), and experiment 2 (judged curvedness). Error bars represent the 95% confidence interval of the regression coefficient.

These data indicate that the integration of disparity, texture, and shading significantly varies according to whether one measures scale-independent or scale-dependent information about surface structure. Such a finding would be difficult to explain if integration occurred on the basis of a common depth-map representation derived from all 3-D-shape cues. However, before we discuss this result in relation to previous studies of perceived shape from individual depth cues, we must be certain that the difference between experiments 1 and 2 did not occur because of the different stimulus sets used in the two experiments. Because the cues in the first experiment differed only in terms of curvedness and those in the second only in terms of shape index, it is possible that our results simply reflect a response bias induced by differential sensitivity to these two aspects of 3-D structure. This possibility will be examined in the third experiment.


5 Experiment 3: Judged shape index and curvedness from disparity, texture, and shading The results from experiments 1 and 2 indicate that the integration of disparity, texture, and shading depends upon the specific aspect of surface structure measured. However, in each experiment the stimulus set varied on different attributes, and thus these results may be attributable to response bias. To account for this possibility, in the third experiment the disparity, texture, and shading cues simultaneously differed in both simulated shape index and curvedness. Varying the cues on both dimensions simultaneously also allowed us to examine other parameterizations of the stimulus space that do not preserve the distinction between scale-independent and scale-dependent surface structure (eg Gaussian curvature and mean curvature).

5.1 Procedure The procedure was identical to that used in the first two experiments, except that the disparity, texture, and shading cues differed in terms of both shape index and curvedness. To accomplish this, on each trial the surfaces depicted by disparity, texture, and shading were randomly assigned from one of the twenty-five shapes depicted in figure 1.

5.2 Results and discussion By using equat ions (5) and (6) we computed the judged shape index and curvedness for each trial. Separate regression analyses were then performed on these t ransformed KX and K2 judgments . Tables 3 and 4 conta in the results of these analyses for shape index and curvedness, respectively. Again we can see tha t the linear model provided an excellent fit of the data . Fur thermore , as in the first two experiments, binocular disparity contr ibuted more to judged 3-D structure than either texture or shading.

To again allow for direct compar isons between the integration of shape index and curvedness we have plotted the s tandardized regression coefficients in figure 6. We can see from these da ta tha t the relative contr ibut ions of disparity, texture, and shading were comparable to those obtained in the first two experiments: binocular disparity contr ibutes significantly more to judged shape index than to curvedness, and texture

Table 3. Results of the multiple regression analysis for the shape-index values computed from the KX and K2 judgments in experiment 3.

Subject Multiple Wsi Wt WA

R2

JFN 0.92 0.704 (0.033) 0.275 (0.028) 0.018 (0.027) JST 0.98 0.802 (0.016) 0.107 (0.016) 0.083 (0.017) JTT 0.99 0.885 (0.010) 0.038 (0.010) 0.037 (0.010) VJP 0.91 0.664 (0.032) 0.295 (0.031) 0.044 (0.031) Note: standard errors appear in parentheses.

Table 4. Results of the multiple regression analysis for the curvedness values computed from the KX and K2 judgments in experiment 3.

Subject Multiple Wst Wx Ws]

R2

JFN 0.99 0.344 (0.013) 0.164 (0.012) 0.100 (0.012) JST 0.98 0.439 (0.021) 0.204 (0.016) 0.051 (0.021) JTT 0.99 0.501 (0.015) 0.103 (0.015) 0.016 (0.015) VJP 0.98 0.249 (0.019) 0.224 (0.021) 0.130 (0.021)

Note: standard errors appear in parentheses.


Stereo Texture Shading Stereo Texture Shading

Figure 6. Bar graphs showing the standardized regression coefficients of binocular disparity (stereo), texture, and shading for judged shape index and judged curvedness computed from the principal curvature adjustments obtained in experiment 3. Error bars represent the 95% confidence interval of the regression coefficient.

and shading contribute more to judged curvedness than to shape index. Because these weights were computed from the same stimuli and on the same trials, we can conclude that the differences between the contributions of stereo, texture, and shading were not due to any bias caused by the stimulus sets.

We have found a disassociation between the relative contributions of three optical sources in the perception of scale-independent (shape index) and scale-dependent (curvedness) surface structure, but would we expect to find a similar result for other parameterizations of our stimuli? Of course, there are an infinite number of 2-D param-eterizations of the local second-order structure of our quadric patches, but not all of these preserve the scale-independent and scale-dependent distinction. We would argue that it is this distinction rather than shape index and curvedness per se that is important in understanding the perception of 3-D structure. One way to examine the importance of the scale-independent and scale-dependent surface structure is to reanalyze the data presented in figure 6 by using a parameterization of the surface that does not separate out these two surface properties. One particularly well-known representation of local surface structure that can be directly computed from the principal curvatures is provided by the Gaussian curvature (K) and the mean curvature (H):

K = KXK2, (7)


Although Gaussian and mean curvature completely specify the local second-order structure of a smooth surface, they do not separate out the scale-independent aspects of surface structure. Consequently, if our previous results were due to a difference between the integration of scale-independent and scale-dependent structure, we would expect to see a different pattern of relative contribution from that shown in figures 5 and 6. The standardized regression coefficients resulting from an analysis of the KX and K2

responses transformed by equations (7) and (8) can be seen in figure 7. The data from figure 7 show the general trend for stereo to contribute more than

either texture or shading to observers' judgments, but we do not see the pattern of relative contribution viewed previously: the standardized regression coefficients for our three cues do not significantly differ between Gaussian curvature and mean curvature. On the basis of this result we would now argue that the distinction between scale-independent and scale-dependent seems to be perceptually relevant and an important factor in the integration of multiple 3-D-shape cues.

jc-p E3 Gaussian curvature mean curvature


JTT

BSpD Stereo Texture Shading

Figure 7. Bar graphs showing the standardized regression coefficients of motion and stereo for judged Gaussian curvature, and adjusted mean curvature. Gaussian curvature and mean curvature were computed from the observers' adjusted KX and K2 . Error bars represent the 95% confidence interval of the regression coefficient.

6 General discussion Understanding how the visual system combines the various optical cues to 3-D structure is an important part of a complete theory of the perception of 3-D form. One aspect of this problem that has been relatively ignored relates to the specific type of structural information the different cues might contribute to our final conscious percept. We have examined the integration of binocular disparity, texture, and shading as a function of


scale-independent (shape index) and scale-dependent (curvedness) surface properties. These two representations were initially chosen because they are based upon coordinate-system-independent aspects of 3-D form (ie K{ and K2). Another reason why we specifically examined the role of scale-dependent and scale-independent information in cue integration was because previous studies indicated that these properties may not be equally specified, within the human visual system, by motion and binocular stereopsis (Tittle and Perotti 1997). The results of the three experiments reported here also indicate that the integration of multiple 3-D-shape cues significantly varies as a function of the specific surface property being measured. This should not be taken to mean that we are arguing that shape index and curvedness have some special perceptual status. Instead we interpret the mean-curvature and Gaussian-curvature analysis in experiment 3 to indicate that the visual system is sensitive to both scale-dependent and scale-independent components of surface structure, but shape index and curvedness simply represent one particular coordinate system that preserves these properties. It is also important to note that, although the specific measures we used in these experiments (ie shape index and curvedness) are computed locally, it would be impossible to infer from the data presented here that observers are explicitly sensitive to local scale-independent and scale-dependent surface structure. Recent results (Phillips and Todd 1996) show that observers' ability to accurately discriminate local surface structure significantly deteriorates if the visible surface region is restricted to 2 deg or less. However, because our task was in essence a global surface match, our data are certainly consistent with the idea that the human visual system is sensitive to global scale-dependent and scale-independent surface structure.

In all three experiments our observers produced the same pattern of results: binocular stereopsis contributed more to judgments of shape index than of curvedness, whereas shading and texture contributed more to judgments of curvedness than of shape index. Considered within the context of computational analyses of shape from stereopsis, texture, and shading these data may seem somewhat paradoxical. Typical theoretical results have suggested that, whereas binocular stereopsis can provide observers with veridical scale-dependent information such as depth (eg Mayhew and Longuet-Higgins 1982), both shading (eg Koenderirik and van Doom 1980; Horn and Brooks 1989) and texture (eg Witkin 1981; Aloimonos 1988) only provide scale-independent information such as local orientation.

On the basis of this theoretical perspective, one would expect that texture and shading should have contributed more to our scale-independent (shape-index) measure than to our scale-dependent (curvedness) measure. However, there are several reasons why our data do not conform to this interpretation. First, the basic phenomenology associated with texture or shading stimuli is not consistent with the idea that we see a one-parameter family of surfaces. Rather observers commonly report seeing a specific surface even when the stimulus only contains shading or texture to define 3-D form. This basic observation implies that our visual system uses some form of heuristic or default scaling to arrive at a unique perceptual interpretation. Furthermore there is no reason a priori that the same estimate of distance used to scale binocular disparities (ie convergence angle or vertical disparities) could not also be used to scale texture and shading. Second, although the majority of computational analyses of shape from shading and texture demonstrate how one could only recover scale-independent information, there are notable exceptions which explicitly relate scale-dependent surface structure to texture and shading information. Pentland (1989) has shown that one can recover depth, up to a scale factor, from only shading information. Further counter examples exist for the case of shape from texture. Specifically, Stevens (1981) has demonstrated that one could compute local depth on the basis of a measure of the characteristic dimension of a surface texture, and Garding (1992) has shown that


the projective distortion of surface textures can specify surface curvature up to a scale factor (ie the perceived distance to the surface). This second group of computational analyses is consistent with a number of psychophysical studies that have provided evidence that human observers can judge scale-dependent properties of surfaces from either shading and texture (eg A Johnston and Passmore 1994a, 1994b) or texture alone (eg Cutting and Millard 1984; Todd and Akerstrom 1987; Cumming et al 1993; E B Johnston et al 1993). Consequently, we consider it plausible that our observers were using some heuristic process to scale the shading and texture information such that these two sources significantly contributed to judged curvedness even in the presence of conflicting binocular-disparity information.

Although some of the manipulations of individual cues presented here have not been examined previously (eg there have been no empirical studies of perceived shape index from texture), several studies of perceived shape from disparity, texture, and shading provide some insight into why we obtained the particular pattern of integration noted above. Recent studies of single 3-D cues have examined the perception of shape index and curvedness from both binocular disparity (deVries et al 1993) and shading (Erens et al 1993; Mamassian et al 1996). Using quadric surfaces simulated by random-dot stereograms, deVries et al (1993) found very high accuracy for both judged shape index and judged curvedness. These results are perfectly consistent with those reported here. We found that binocular stereopsis contributed more than shading and texture for both shape index and curvedness.

The ability of observers to perceive shape index from shading is not as obvious as the results reported for binocular stereopsis. Erens et al (1993) found no evidence that observers could reliably judge shape index on the basis of Lambertian-shaded quadric surfaces, but Mamassian et al (1996) obtained a high degree of accuracy when they required observers to classify regions on a solid object as either elliptic or hyperbolic. The data from our study seem to fall somewhere in between these two earlier shading results. We found that shading provided a statistically significant contribution to judged shape and curvedness, but the magnitude was considerably less than that resulting from stereo and texture. One important difference between the stimulus conditions used by Mamassian et al (1996) and those used both by us and Erens et al (1993) concerns the presence of a self-occluding contour. Because Mamassian et al (1996) simulated a solid 3-D object, their displays had visible self-occluding contours which have been shown to provide important context for the perception of shape from shading (Todd and Reichel 1989). We used quadric surface patches such as those studied by Erens et al (1993) which do not have self-occluding contours. Thus, we would expect a greater contribution of shading to judged shape from stimuli with visible occluding contours, but that remains an issue for future research.

It is a little more difficult to compare our results with earlier research regarding perceived 3-D structure from conflicting cues because these studies all considered only scale-dependent variations of 3-D shape. However, to the extent that our stimulus conditions overlap with those of previous researchers we find general agreement between their findings and those reported here. The first study to examine shape from conflicting shading and stereo displays was conducted by Biilthoff and Mallott (1988). They found that edge-based stereo contributes significantly more to observers' depth judgments than either shading alone or disparate shading (the difference between the smooth shading patterns in the two eyes). The stimulus manipulations made by Bulthoff and Mallott (1988) were the equivalent of curvedness variations in our paradigm. Thus, their finding that shading provides a significant but small contribution to judged depth is consistent with the results reported here for our curvedness conditions.

Binocular stereopsis has also been found to dominate but not veto judged depth when placed in conflict with texture. E B Johnston et al (1993) examined the judged


depth of cylindrical surfaces in which the depth-to-height ratio specified by texture and stereo were in conflict. These experiments are the equivalent of our curvedness manipulations for cylindrical surfaces (shape index = 0.5) as shown in figure 1. As we have reported here, they found that stereo did not simply veto the texture cue, but instead the combination could be described by a weighted linear model. For all of their stimulus conditions E B Johnston et al (1993) also reported that stereo had a significantly greater weight than texture. Perhaps the most interesting aspect of their experiment is that the weight assigned to texture increased as the viewing distance was increased. As E B Johnston et al (1993) noted, this implies that as scaling information becomes weaker the texture cues begin to contribute more to the judged percept. We would argue that a similar phenomenon occurs for scale-dependent versus scale-independent components of surface structure.

Up to this point we have described how binocular disparity, texture, and shading contribute to judged 3-D structure. However, these data are also useful when considering the more general issue regarding the plausibility of certain classes of shape-integration models. We believe they are particularly relevant to the question of what type of information is being integrated from the various sources of optical information. Current theories of cue integration are based upon the assumption that there is a single common representation shared by all sources before they are integrated (Biilthoff and Yuille 1991; Cutting and Vishton 1995; Landy et al 1995). A common choice for this representation is the depth map, but this representation does not easily allow the integration of ordinal cues such as occlusion which have been shown to play an important role in the integration of multiple depth cues (Braunstein et al 1986). Cutting and Vishton (1995) have suggested that one way to get around this difficulty is to first compute an ordinal representation of depth from all available sources. However, we would argue that, regardless of the nature of the representation, our data seem to be inconsistent with any model which assumes that all cues contribute to a single representation.

If we consider an integration model that assumes a common initial representation, it can be seen that certain inconsistencies with the data reported here result. If we assume that the final integration is linear, a supposition consistent with our data on the basis of the high R2 values shown in tables 1-3, the relative contributions of binocular stereopsis, texture, and shading are determined by the 3-D weight vector Wt> ^t^ ^sh)- Such an assumption is consistent with the results reported here, because after integrating our three cues to get a depth map we can simply compute local curvedness and shape index as shown in equations (3)-(6). However, if we consider the relative contribution of binocular disparity, texture, and shading to these computed shape-index and curvedness values a problem arises. If these measures are computed from the same depth map, it follows that they should each have the same weight vector (FFst, Wt, Wsh) which describes the relative contribution of binocular disparity, texture, and shading. We did not find evidence to support this prediction because the relative contribution or weight of each of our three sources was significantly different for shape index and curvedness.

As an alternative to the class of models that assume a single initial representation we would argue that our data are more consistent with an approach in which the processing of each cue results in multiple outputs (eg depth, curvedness, and shape index). In this view the depth map is no longer a special first step in the computational process but rather just another possible local representation. Therefore, the difficulty from predicting the same weight vector for the curvedness and shape-index values does not occur. Such models can easily accommodate a unique weight vector for each of the local 3-D properties being computed: such as depth (Z>st, Z)t,Z>sh), curvedness (Cst,Ct, Csh), or shape index (Sst,St,Ssh).


The preceding discussion depended in part upon the assumption of comparable scaling for perceived binocular stereopsis, texture, and shading. Although we did not test this assumption explicitly, we will now consider how our interpretation of the weight values would change if the scaling of the three cues was not approximately equal. If one considers the quadric surface specified in equation (1), it can be seen that the values KX and K2 could be scaled unequally for binocular disparity (cst), texture (dt), and shading (<7sh). If it were the case that <rst = at = <rsh, we could interpret the weights shown in figures 5 and 6 as the relative contributions of binocular disparity, texture, and shading to judged curvedness and shape index.

The most important consequence of differences between the scaling of these cues occurs because one of our measurements of 3-D structure is scale independent (shape index) and the other is scale dependent (curvedness). It can be shown that scale plays no role in specifying the local shape index because taking the ratio of the principal curvatures KX and K2 , as shown in equation (4), cancels out the scale applied to each principal curvature. Thus, it turns out that the weights for the curvedness judgments are scale dependent, but the weights obtained for shape-index judgments are not. The consequences of this can be seen by considering two new variables which equal the relative contribution of disparity and texture to shape index (Rs = WsXS/WiS) and curvedness (Rc = W%tc/WxC\ where WstS and Wts equal the weights for the stereo and texture contribution to judged shape index, and Wstc and WiC equal the weights for the stereo and texture contribution to judged curvedness (as shown in figures 5 and 6). Because Rs is scale dependent and Rc is not, we would expect the following simple relation to hold if the relative contribution of stereo and texture were actually the same for shape index and curvedness: Rs = Rc(ast/at). In experiment 3, the average value of the scaling of disparity relative to texture (o"st/o-t) would have to equal approximately 3.7 in order for the contribution of stereo and motion (Rs and Rc) to be equal for judgments of shape index and curvedness. Consequently, from our data alone we can not rule out the possibility that the weight differences shown in figures 5 and 6 resulted from unequal scaling of binocular disparity, texture, and shading.

Although it may be possible to achieve different relative contributions from a single initial-representation model, we would remind the reader of the recent results by A Johnston and Passmore (1994a, 1994b) and Norman and Todd (1996) strongly suggesting that observers do not directly recover local depth from surfaces specified by disparity, shading, and texture. Further confirmation of these findings comes from Koenderink and van Doom (1995), who have reported that observers can not accurately compare depths of widely separated surface regions but they can accurately compare orientations of widely separated regions. At the very least, these studies and the data reported here indicate that the depth map is not a viable representation upon which to base the integration of multiple 3-D-shape cues. The type of representational framework that may more effectively account for data such as those presented here has been proposed recently by Stevens (1995) and involves a five-dimensional shape space containing both intrinsic (eg local-curvature) and extrinsic (eg local-orientation) properties of surfaces.

Acknowledgements. We would like to thank Del Lindsey and James Todd for helpful comments and discussions regarding the issues described in this paper.

References Aloimonos J, 1988 "Shape from texture" Biological Cybernetics 58 345-360 Braunstein M L, Andersen G J, Rouse M W, Tittle J S, 1986 "Recovering viewer-centered depth

from disparity, occlusion, and velocity gradients" Perception & Psychophysics 40 216-224 Bruno N, Cutting J E, 1988 "Minimodularity and the perception oflayout" Journal of Experimental

Psychology: General 111 161 -170


Buckley D, Frisby J P, 1993 "The integration of stereo, texture, and outline cues in the shape perception of three-dimensional ridges" Vision Research 33 919 - 933

Biilthoff H H, Mallott H, 1988 "Integration of depth modules: stereo and shading" Journal of the Optical Society of America A 5 1749 -1758

Bulthoff H H, Yuille A L, 1991 "Bayesian models for seeing shapes and depth" Comments Theoretical Biology 2 283-314

Cumming B G, Johnston E B, Parker A J, 1993 "Effects of different texture cues on curved surfaces viewed stereoscopically" Vision Research 33 827-838

Cutting J E, Millard R T, 1984 "Three gradients and the perception of flat and curved surfaces" Journal of Experimental Psychology: General 113 198-216

Cutting J E, Vishton P M, 1995 "Perceiving layout and knowing distances: the integration, relative potency, and contextual use of different information about depth", in Handbook of Perception and Cognition volume 5 Perception of Space and Motion Eds W Epstein, S Rogers (San Diego, CA: Academic Press)

Damme W J M van, Oosterhoff F H, Grind W A van de, 1994 "Discrimination of 3-D shape and 3-D curvature in active vision" Perception & Psychophysics 55 340-349

Erens R G F, Kappers A M, Koenderink J J, 1993 "Perception of local shape from shading" Perception & Psychophysics 54 145 -156

Foley J D, Dam A van, Feiner S K, Hughes J F, 1990 Computer Graphics: Principles and Practice (Reading, MA: Addison-Wesley)

Garding J, 1992 "Shape from texture for smooth curved surfaces in perspective projection" Journal of Mathematical Imaging and Vision 2 327 - 350

Gibson J J, 1950 The Perception of the Visual World (Boston, MA: Houghton-Mifflin) Gillam B, Chambers D, Russo T, 1988 "Postfusional latency in stereoscopic slant perception and the

primitives of stereopsis" Journal of Experimental Psychology: Human Perception and Performance 14 163-175

Horn B K P, Brooks M J, 1989 Shape from Shading (Cambridge, MA: MIT Press) Johnston A, Passmore P J, 1994a "Shape from shading I: Surface curvature and orientation"

Perception 23 169-189 Johnston A, Passmore P J, 1994b "Independent encoding of surface orientation and surface

curvature" Vision Research 34 3005-3012 Johnston E B, Cumming B G, Landy M S, 1994 "Integration of stereopsis and motion shape cues"

Vision Research 34 2259-2275 Johnston E B, Cumming B G, Parker A J, 1993 "Integration of depth modules: stereopsis and

texture" Vision Research 33 813 - 826 Koenderink J J, 1990 Solid Shape (Cambridge, MA: MIT Press) Koenderink J J, Doom A J van, 1980 "Photometric invariants related to solid shape" Optica Acta

27 981-996 Koenderink J J, Doom A J van, 1995 "Relief: pictorial and otherwise" Image and Vision Computing

13 321-333 Kreyszig E, 1959 Differential Geometry (Toronto: University of Toronto Press) Landy M S, Maloney L T, Johnston E B, Young M, 1995 "Measurement and modeling of depth cue

combination: in defense of weak fusion" Vision Research 35 389-412 McKee S P, Levi D M, Bowne S F, 1990 "The imprecision of stereopsis" Vision Research 30

1763-1779 Mamassian P, Kersten D, Knill D C, 1996 "Categorical local-shape perception" Perception 25

95-107 Mayhew J E, Longuet-Higgins H C, 1982 "A computational model of binocular depth perception"

Nature (London) 297 376-378 Norman J F, Todd J T, 1996 "The discriminability of local surface structure" Perception 25 381-398 Norman J F, Todd J T, Phillips F, 1995 "The perception of surface orientation from multiple sources

of optical information" Perception & Psychophysics 57 629 - 636 Pentland A, 1989 "Shape information from shading: A theory about human perception" Spatial

Vision 4 165-182 Perotti V J, Todd J T, Lappin J, Phillips F, 1995 "Perceived shape index and curvedness from

constant flow fields" Investigative Ophthalmology and Visual Science, Supplement 36 361 Phillips F, Todd J T, 1996 "Perception of local three-dimensional shape" Journal of Experimental

Psychology: Human Perception and Performance 22 930 - 944 Rogers B J, Collett T S, 1989 "The appearance of surfaces specified by motion parallax and

binocular disparity" Quarterly Journal of Psychology A 41 697-717


Rogers B J, Graham M E, 1983 "Anisotropics in the perception of three-dimensional surfaces" Science 221 1409-1411

Stevens K A, 1981 "The information content of texture gradients" Biological Cybernetics 42 95-105 Stevens K A, 1995 "Integration by association: combining three-dimensional cues to extrinsic

surface shape" Perception 24 199 - 214 Stevens K A, Lees M, Brookes A, 1991 "Combining binocular and monocular curvature features"

Perception 20 425 - 440 Tittle J S, Braunstein M L, 1993 "Recovery of 3-D shape from binocular stereopsis and structure

from motion" Perception & Psychophysics 54 157 -169 Tittle J S, Perotti V J, 1997 "The perception of shape and curvedness from binocular disparity,

texture, and shading" Perception & Psychophysics 54 157 -169 Tittle J S, Perotti V J, Norman J F, 1997 "The integration of binocular stereopsis and structure-

from-motion in the discrimination of noisy surfaces" Journal of Experimental Psychology: Human Perception and Performance 23 1035 -1049

Tittle J S, Todd J T, Perotti V J, Norman J F, 1995 "The systematic distortion of perceived 3-D structure from motion and binocular stereopsis" Journal of Experimental Psychology: Human Perception and Performance 21 663 - 678

Todd J T, Akerstrom R, 1987 "Perception of three-dimensional form from patterns of optical texture" Perception & Psychophysics 13 242-255

Todd J T, Reichel F D, 1989 "Ordinal structure in the visual perception and cognition of smoothly curved surfaces" Psychological Review 96 643 - 657

Ullman S, 1979 The Interpretation of Visual Motion (Cambridge, MA: MIT Press) Vries S C de, Kappers A M L, Koenderink J J, 1993 "Shape from stereo: a systematic approach

using quadratic surfaces" Perception & Psychophysics 53 71-80 Witkin A P, 1981 "Recovering surface shape and orientation from texture" Artificial Intelligence

17 17-45


APPENDIX: Generating conflicting texture and binocular-disparity stimuli A conflicting texture and binocular-disparity display can be generated by drawing a texture element in the tangent plane to the surface, and then projecting the circle back onto the surface by using the mapping

I y J ~ I y I , (Al) \zj \f(x9y)J

where f(x, y) is the z-coordinate of the binocular-disparity-defined quadric surface. To construct a circular texture element in the tangent plane, we start with a circle in the x, y plane, whose points are given by the coordinates

/Rcos6\ RsinO = RxcosO + RysinO, (A2)

Vo / where x and y are unit vectors along the x-axis and y-axis, and R is the radius of the circle. We next rotate the coordinate system, mapping x, y, z into rl9 r2, r3 to place the circle in the tangent plane, and translate the center of the circle to some point

(A3)

on the surface. Thus the circle in the tangent plane can be represented parametrically as the set of points

{r0 + RrY cos 9 + Rr2 sin 9} . (A4)

The unit vectors rx and r2 may be any mutually orthogonal unit vectors in the tangent plane, which means that they must be orthogonal to the local unit normal

Such a set of unit vectors can be constructed by using the cross product:

ri = (%)=-{"= 1 Jl ) (A6)

is a unit vector perpendicular to n (and, incidentally, to y\ and

r2 = ( % | = J ^ L = l- — ( ti+nl | (A7) \rl) l"^'! [«X+«M2 + K2+«/)2]'/2V-nynz J

is orthogonal to both n and rx. Last, using the mapping defined in equation (Al) above we obtain the x and y coordinates (we are going to use the binocular-disparity-specified surface to supply the z coordinate) of the circle projected onto the surface:

( R (rlx cos 9 + r2x sin 9) \ i*(r lvcos0 + r2>,sin0) . (A8)

© 1998 a Pion publication printed in Great Britain

n(x0,y0) = ( ny | = — — 7 - r - — — ^ ( -K2y0 ) . (A5)

Documents

The perception of scale-dependent and scale-independent ...€¦ · Perception, 1997, volume 26, pages 147-166 The perception of scale-dependent and scale-independent surface structure