Imagery slidesImagery and MemoryMemory Examples: Dual Code TheoryTo recall Y you must first recall XWindows, doorknob, glasses, other facial features, global-to-localBut: Something like the same thing happens in recall of alphabet letters and many other memorized listsImageability rating are more effective than frequency of occurrence or frequency of co-occurrence in paired-associates learning.Vision is clearly involved when images are superimposed onto visionMany experiments show that when you project an image onto a display the image acts very much like a superimposed displayShepard & Podgorny (paper folding task)Interference effects (Brooks)Controvercial Perky effect: Perception or response bias?3Project an image onto a perceived formBrooks spatial interference study
Respond by pointing to symbols in a table or by saying the words left or right5Perception or attention effects?Many impressive imagery effects can be plausibly attributed to attentionBisiach widely-cited finding on visual neglectBartolomeo, P., & Chokron, S. (2002). Orienting of attention in left unilateral neglect. Neuroscience and Biobehavioral Reviews, 26(2), 217-234.Dulin, D., Hatwell, Y., Pylyshyn, Z. W., & Chokron, S. (2008). Effects of peripheral and central visual impairment on mental imagery capacity. Neuroscience and Biobehavioral Reviews, 32(8), 1396-1408.Does neglect require vision?Chokron, S., Colliot, P., & Bartolomeo, P. (2004). The role of vision in spatial representations. Cortex, 40, 281-290.
We can to some extent control our attended regionFarah, M. J. (1989). Mechanisms of imagery-perception interaction. Journal of Experimental Psychology: Human Perception and Performance, 15, 203-211.
Is an image being projected onto a percept, or just a selective attention?7Shepard & Podgorny experimentBoth when the displays are seen and when the F is imagined, RT to detect whether the dot was on the F is fastest when the dot is at the vertex of the F, then when on an arm of the F, then when far away from the F and slowest when one square off the F.
8Similarities between perception of visual scenes and perception of mental imagesJudgments from mental imagesShape comparisons (of states: Shepard & Metzler)Size comparisons (Weber fraction or ratio effect)What do they tell us about the format of images?But this applies to nonvisual properties (e.g., price, taste)More demonstrations of the relation between vision, imagery (and later action)Images constructed from descriptionsThe D-J example(s)Perception or inference/guessing But there are even more persuasive counterexamples we will see laterThe two-parallelogram exampleAmodal completionReconstruals: Slezak10Dynamic imageryImagining actions: Paper Folding
Time to judge whether (a)-(b) or (b)-(c) are the same except for orientation. Time increases linearly with the angle between them (Shepard & Metzler, 1971) 12What do you do to judge whether these two figures are the same shape?
When you make it rotate in your mind, does it seem to retain its rigid 3D shape without re-computing it?Is this how the process looked to you?
13Rotation is neither holistic nor impenetrable
Mental rotation the real story In mental rotation the phenomenology motivates the theory of rotation but what the data actually show is that,Mental rotation is only found when the comparison figures are enantiomorphs or if the difference between figure pairs can only be expressed in figure-centric coordinates eg. they are 3D mirror-imagesNo rotation occurs if the figures have landmarks that can be used to identify the relations among their parts.Records of eye movements show that mental rotation is done incrementally: It is not a holistic rotation as often reported. If fact even the phenomenology is not of a smooth continuous rotation.The rate of rotation depends on the conceptual complexity of both the figure and comparison task so that, at least, is not a result of the architecture (Pylyshyn, 1979). There are even demonstrations that it depends on how the subject interprets the figure (Kosslyn, 1994).1515Mental ScanningHundreds of experiments have now been done demonstrating that it takes longer to scan attention between places that are further apart in the imagined scene. In fact the relation is linear between time and distance.These have been reviewed and described in:Denis, M., & Kosslyn, S. M. (1999). Scanning visual mental images: A window on the mind. Cahiers de Psychologie Cognitive / Current Psychology of Cognition, 18(4), 409-465.16Studies of mental scanningDoes it show that images have metrical space?
Does this show that images are spatial, or have spatial properties, or that they preserve metrical spatial properties? (Kosslyn, S. M., T. M. Ball, et al. (1978). "Visual images preserve metric spatial information: Evidence from studies of image scanning." Journal of Experimental Psychology: Human Perception and Performance 4: 46-60.
17The idea of images being in some sense spatial is an interesting and important claimI will discuss this claim at some length later because it reveals a deep and all-consuming error that runs through all imagery theorizing by psychologists, neuroscientists and philosophers. This is in addition to the errors I discussed earlier: The idea that subjects understand the task of imagining something to be the task of pretending they are seeing it, and the idea that certain properties of the world are properties of the image (the intentional fallacy)Constructing an imageWhat determines what the image is like when it is constructed from memory or from knowledge?After constructing an image can you see novel aspects of the imagined situation?ExamplesImagine seeing these events unfoldingYou hit a baseball. What shape trajectory does it trace? It is coming towards you: Where would you run to catch it? If you have ever played baseball you would have a great deal of tacit knowledge of what to do in such (well studied) cases.You drop a rubber ball on the pavement. Tap a button every time it hits the ground and bounces. Plot height vs time.
Drop a heavy steel ball at the same time as you drop a light ball (a tennis ball), e.g., from the leaning tower of Pisa. Indicate when they hit the ground. Repeat for different heights.Take a clear glass containing a colored liquid. Tilt it 45 to the left (counter-clockwise). What is the orientation of the liquid?Examples to probe your intuition and your tacit knowledgeTime since dropheightWhat is responsible for the pattern shown here?20What color do you see when two color filters overlap??21Where would the water go if you poured it over a full beaker of sugar?
Is there conservation of volume in your image? If not, why not?2222Seeing Mental ImagesDo images have size? Can we say that one image is larger than another?If so, what properties do we expect the smaller/larger image to have?
Do mental images have size?Imagine a very small mouse. Can you see its whiskers? Now imagine a huge mouse. Can you see its whiskers?
25Do this imagery exercise:Imagine a parallelogram like this oneNow imagine an identical parallelogram directly below this oneConnect each corner of the top parallelogram with the corresponding corner of the bottom parallelogramWhat do you see when you imagine the connections?Did the imagined shape look (and change) like the one you see now?26Slezak figures
Pick one (or two) of these animals and memorize what they look like. Now rotate it in your mind by 90 degrees clockwise and see what it looks like.27Slezak figures rotated 90o
28SpaceP 29Images and the representation of spatial propertiesWe need to understand what it could mean for a representation to be spatial.At the very least it must mean that there are constraints placed on the form of the representation that do not apply when the representation is not spatial.Studies of mental scanningDoes it show that images have metrical space?
Does this show that images are spatial or have spatial properties or that they preserve metrical spatial properties? (Kosslyn, S. M., T. M. Ball, et al. (1978). "Visual images preserve metric spatial information: Evidence from studies of image scanning." Journal of Experimental Psychology: Human Perception and Performance 4: 46-60.
31The idea that images are in some sense spatial is an interesting and important claimI will return to this claim later because it reveals a deep and ubiquitous error that runs through most (all?) imagery theorizing by psychologists, neuroscientists and philosophers. This is the error of mistaking descriptive adequacy with explanatory adequacy. Lets call this conflating, the missing constraint error.This is in addition to the two errors I discussed earlier: Ignoring the fact that the task of imagining something is actually the task of pretending you are seeing it, and The mistaken assumption that certain properties of the world are properties of the image (the intentional fallacy)Connecting Images and Motor actionsImages and visual-motor phenomenaS-R Compatibility / Simon effect Finkes imagined wedge-prism goggles Harrys subitizing-by-pointing
Another chapter in the imagery debate:The interaction of images with vision and motor controlOne of the properties of mental images that makes them appear spatial is that they connect in certain ways not only with vision, but also with the motor system:We can point to things in our image! We can project our images onto perceived space even space perceived in different modalities. I believe that this observation is the key to understanding the spatial character of images.This projection does not require a picture to be projected, only the location of a small number of features. Over the past few decades I have been studying a mechanism called a visual index, or a FINST, that is well suited for this task.Important idea! 34Both vision and visual imagery have some connection to the motor systemThere are a number of experiments showing the close connection between images and motor control*You can get Stimulus-Response compatibility effects between the location of a stimulus in space and the location of the response button in space, Ronald Finke showed that you could get adaptation with the position of the misperceived hand that was similar to adaptation to displacing prism goggles,Both these findings provide support for the view that the spatial character of images comes from something being projected onto a concurrently perceived scene and then functioning much as objects of perception.This is the main new idea in Chapter 5 of Things & Places)an imageimagined 35S-R Compatibility effect with a visual display:The Simon effect: It is faster to make a response in the direction of an attended objects than in another directionResponse for A is faster when YES in on the left in these displays
36S-R Compatibility effect with an imagined displayRT is faster when the A is recalled (imagined) as being on the leftThe same RT pattern occurs for a recalled display as for a perceived one
37Recall the studies of mental scanning
But the way we compute the time it takes to scan across an image is by imagining something moving across the real perceived display. Without this display, we could not use our time-to-collision computation to compute the time to cross various distances on the image because there are no actual distances on the image! (Pylyshyn & Cohen, 1999)
We showed that the image scanning effect is Cognitively Penetrable
Does this result show that images have spatial properties?Does this result show that images have metrical properties?38Using a concurrently perceived room to anchor FINSTs tagged with map labels
39The Spatial character of imagesWhat does it mean to say that images are spatial? It means that certain constraints hold among spatial measures (e.g., axioms of geometry and measure theory, such as triangle inequality, symmetry of distances, Euclidean axioms, Pythagoras theorem}That certain constraints hold among distances, that certain relations can be defined among these distances (e.g., between, farther than), that Newtonian Physics holds between the terms that are used in explanations (e.g., distances and time).That mental images and motor control interact with one another to some degree so you can point to objects in your image.Certain visual-motor reflexes are automatic or preconceptual They are computed within the encapsulated Visual Module Preconceptual motor control is not sensitive to visual illusions, relative to control that is computed by the cognitive (seeing as) system.
Evidence for a literal spatial display in the brainI will discuss the proposal that V1 is the imagery display in the brain. But since the conclusion will be that it is not, lets look at other options.The problem is to explain such phenomena as the scanning effect or the size effect without assuming a physical display.The main alternative to a spatial display is something called a functional space. This proposal was introduced by Kosslyn in his characterization of the depictive nature of the image representation.
Mental images as depictive representationsA depictive representation is a type of picture, which specifies the locations and values of configurations of points in a space.The space in which the points appear need not be physical but can be like an array in a computer, which specifies spatial relations purely functionally. That is, the physical locations in the computer of each point in an array are not themselves arranged in an array; it is only by virtue of how this information is read and processed that it comes to function as if it were arranged into an array.Depictive representations convey meaning via their resemblance to an object.When a depictive representation is used, not only is the shape of the represented parts immediately available to appropriate processes , but so is the shape of the empty space [and] one cannot represent a shape in a depictive representation without also specifying a size and orientation.
42Form vs Content of imagesAs in earlier discussion, one must be careful in distinguishing form from content. We know that there is a difference between the content of images and the content of other (nonimaginal) thought: Images concern sensory appearances while propositions can express most* other contents.In attributing a special form of representation to images one should ask whether some symbolic system (e.g., sentences of LOT) would not do. Simplicity (Occams Razor) would then prefer a single format over two, especially if the one format is essential for representing thoughts and inferences [Fodor, J. A. and Z. W. Pylyshyn (1988). "Connectionism and cognitive architecture: A critical analysis." Cognition 28: 3-71.]The most promising contents that might require different forms of representation are those that es...