A prospectus for instructional research on visual literacy

A Prospectus for Instructional Research on Visual Literacy

W, Howard Levie

SENSORY MODAL1TIES VS.

SYMBOLIC MODALITIES

Among the areas that have been mentioned as contributing to the study of visual literacy are aesthetics, anatomy of the eye, body language, cognit ive psychology , communica t ion theory, cultural anthropology, instructional technology, mental imagery, neurophysiology, perceptual development , psycholinguistics, semantics, and visual perception. While an interdisciplinary approach can be beneficial to a scholarly area during its early phases of development, such eclectic periods must evolve into periods with clear conceptual foci if a field is to have an identity of its own.

In this prospectus I attempt to define one such focus for the study of visual literacy. My suggestions are directed primarily to those concerned with instructional problems in visual literacy; they are less germane to those concerned with such problems as the impact of mass media on society and the aesthetic function of pictorial media. One of the most apparent implications of the term "visual literacy" is that it has something to do with visible (see-able) stimuli. Therefore, one logical place to begin an analysis is to question the nature of these stimuli. The most widely pub- licized definition of visual literacy is one that was tentatively offered by Jack Debes at the first convention of what is now the International Visual Literacy Association. Debes (1970)

W. Howard Levie is associate professor, Instructional Systems Technology, at Indiana University, Bloomington, IN 47401.

ECTJ, VOL. 26, NO. 1, SPRING 1978 25

EDUCATIONAL COMMUNICATION & TECHNOLOGY SPRING 1978 : 26

stated that the visually literate person can interpret "the visual actions, objects and/or symbols, natural or man-made, that he encounters in his environment" (p. 14). This definition, if taken literally, not only circumscribes an unmanage- ably wide and diverse domain, but also includes the ability to read words. It is, of course, this latter abi l i ty- -verbal l i teracy--with which visual literacy is often contrasted. The key problem with Debes' definition is not that it includes too many or inappropriate stimuli, but that it defines the stimuli of interest in terms of a sensory modality rather than a symbolic modality. The type of stimuli of central concern to the study of visual literacy is pictures. This is not because pictures are visual (can be seen) but because pictures are symbols and, more specifically, symbols that are neither words nor some other kind of digital symbol. (See Figure 1.) Thus it is symbolic behavior, not sensory behavior, that is of most interest to those in the visual literacy movement. In both cases the stimuli may be the same, but the kinds of internal (mental) processes involved are different in important ways.

What kinds of mental processes may be involved in interpreting pictures? 1 Alan Paivio (1971) proposes that human

FIGURE 1

Symbolic and Sensory Modalities

~ .~> "O O

O

Symbolic Modalities

Digital Signs

Printed words

Numbers

Semaphore, etc.

Spoken words

Morse code

Alarms, etc.

Iconic Signs

Pictures

Statues

Gestures, etc.

Sound effects

Onomatopoeia

Music (?)

Nonsign Stimuli

Natural objects

Man-made nonsign stimuli

Naturally occurring and man-made noise

1Prerequisite to the ability to interpret pictures are certain fundamental understandings about symbol use. For example, a person must have the concept that a picture is not just markings on a surface but is rather a stimulus that stands for something else.

INSTRUCTIONAL RESEARCH ON VISUAL LITERACY : 27

FIGURE 2

A Model of the Iconic Mode

Internalization,

Perceptual ~ / Representation Mental (Pictures) Representation (Imagery)

,~X Externalization

A CONCEPTUAL CORE: THE

ICONIC MODE

information processing involves two systems: a verbal symbolic system and an imaginal symbolic system. The verbal system is specialized for processing temporal, sequential information of a relatively abstract nature. The imaginal system is specialized for processing spatial, s imultaneous information of a relatively concrete nature. While the two systems are richly interconnected, each can function independent ly . It is the independen t operations of the imaginal system, called "menta l imagery" or "visual th inking," that form the internal link of what I will call the "iconic mode . " As shown in Figure 2, the iconic mode consists of an external, perceptual representation (pictures), an internal, mental rep- resentat ion (imagery), and the connections be tween them. These connections are processes of (a) internalizing the mean- ings available in pictures in the form of mental imagery, and (b) externalizing mental imagery in the form of pictorial representations. These processes are roughly analogous to (a) reading and (b) writing in the verbal mode.

At the outset let me concede that there are both conceptual and operational difficulties with this model. Conceptually, some people argue that, compelling though the subjective experience of mental imagery may be, the observational data do not warrant the inclusion of this "mental is t ic" concept in scientific theory. 2 In my view, the operational difficulties with

2Research data to support claims for the existence of mental processes of the kind referred to as "mental imagery" are being produced. For example, Shepard and Metzler (1971) showed that the time required to perform the mental operation of rotating a geometric figure is linearly correlated with the amount of rotation requested. Just and Carpenter (1976) showed that eye movement patterns of subjects engaged in these mental rotation tasks are suggestive of the imaginal manipulation of perceptual spatial information. For recent debate concerning the scientific status of the concept of mental imagery, see Pylyshyn (1973), Kosslyn and Pomerantz (1977), and Fleming (1977).


The Iconic Mode and Symbolic Codes

the model are more severe. How can the imaginal processes evoked as a result of seeing pictures be studied? Drawing pictures, the most direct means of externalizing mental imagery, is not very satisfactory since few people are skilled artists and even their drawings are but rough approximations of their internal imagery. (Shepard [1976] describes a few technically difficult possibilities such as the methods used by police artists to reconstruct the likeness of a suspect based upon the directions of eye witnesses.) It appears that researchers must be content with second-level indicators of imaginal activity such as reaction times, ability to perform certain spatially-relevant tasks, and verbal reports of mental imagery.

The fact that the model deals only with images is not meant to imply that words and verbal processes are irrelevant or unimportant to the iconic mode. Indeed, verbal and imaginal processes are often mutually stimulating. Pictures are fre- quently interpreted in verbal terms, and words may arouse mental imagery. In instructional practice, pictures are rarely used in the absence of words, and learning from pictures is usually tested verbally. In spite of the importance of words and verbal processes, I propose that for the purpose of prob- ing the issues that are central to people concerned with instructional problems in visual literacy, this model of an iconic mode can serve as a conceptual core. What is the nature of the information available in pictures? As Salomon (1974) notes, media can be represented as consisting of messages (subject-matter content), technologies of trans- mission (roughly, the tools used), and symbolic systems. The symbolic system of a medium is a set of symbolic codes that describe the relationship between some aspect of a picture and its internal representation. The elements of a particular symbolic code are (a) a type of pictorial cue, (b) the mental imagery this cue evokes, and (c) the mental operations required to make the translation from the pictorial cue to the mental representation. (In many cases it may be clearer to think of (b) and (c) as synonymous so that a code may be thought of as the external and internal representations of one operation.) Thus it is the symbolic codes used by pictorial media that reveal the nature of the iconic mode.

What are the symbolic codes of pictorial media? The problem can be studied at various levels of abstraction. One can study the codes employed by a particular medium, such as

INSTRUCTIONAL RESEARCH O N VISUAL LITERACY : 29

A Continuum of Iconic Codes

those used in maps or cartoons, or the codes common to groupings of media, such as still representative pictures (drawings, photographs, etc.) or logical pictures (schematic diagrams, flow charts, etc.; see Knowlton, 1966). Or one can study the codes related to various pictorial stimulus attributes (color, motion, etc.). At the highest level of abstraction, one can a t t emp t to d i scover codes that are sha red by all p ic tures-- the most fundamental codes that distinguish the iconic mode from the verbal mode. As Arnheim (1974) states:

Remote though the wall paintings of the ice age may be from the television screen, the eye looks at pictures in both instances; and with all the differences between writing hieroglyphs on papyrus and setting type by computer, the psychology of reading has not changed substantially. (p. 181)

Thus a continuum of iconic codes can be conceived, the most superordinate codes being applicable to all pictorial media and the most subordinate codes being properties of only one particular medium or pictorial stimulus attribute.

As shown in Table 1, the topmost part of the continuum consists of codes that specify the most fundamental relationships between external (perceptual) images and their corresponding internal (mental) representations. What can we say about the nature of these relationships?

It is no longer seriously proposed that the structural ar- rangement of the stimuli in a picture has an exact internal representation. That is, seeing a picture of an object does not result in a set of neurons being fired off in the brain forming a pattern like a line drawing or the half-tone dots in a newspa- per photograph. Such a concept, among other shortcomings, would fail to account for the fact that we can identify an object in a wide range of structurally different pictorial representations. According to Roger Shepard (Shepard & Chipman, 1970) the critical aspect of the similarity between the two images is functional rather than structural. The mental images evoked by various pictures of an object function as if they were equivalent to each other and to the percept of the actual object as well. Shepard explains:

Despite the practically unlimited range and diversity of possible internal representations, we can readily assess within ourselves the degree of functional relation between any two by a simple, direct judgment of subjective similarity. Moreover, we can do this even though (a) we have never before compared the two representations


TABLE 1

Examples of Iconic Codes

Types of Cues Provided Corresponding Pictorial Media by the Pictorial Mental Using the Code Stimulus Operations

All representative pictures

Many pictures

Media groups (e.g., film & television)

logical pictures

Media types, e.g., maps

cartoons

Cubist paintings

boundary line information (e.g., edges)

linear perspective, overlap, gradients

disequilibrium

continuity editing (e.g., screen direction)

camera movement (e.g., pans, dollies)

labeled boxes and connecting lines

contour lines

implied motion cues

showing an object from multiple points-of-view

figure-ground separations

third dimension discriminations

movement downward

continuity of action between scenes

changing viewpoints (e.g., mental rotations)

elements and relationships between them

elevation of land surface

movement of objects

simultaneous third dimensional imagery

in question, and even though (b) we may be unable to communicate anything about the absolute nature of either of the two representations taken separately. (p. 2)

Thus, some kind of functional equivalenCe be tween images that defies exact verbal description is at the root of our ability to interpret pictures. This is the sense in which the information most characteristic of the iconic mode is nontranslatable into the verbal mode and w h y one picture is "wor th" more than any number of words.

What are the most pervasive iconic codes of pictorial media and what pictorial cues are involved? Among the cues in this class are apparent ly those that define boundar ies and lead to the f igure-ground discriminations required for the identification of objects. Hagen (1974) reviews research indicating that the boundary line information available in simple line drawings is often sufficient to allow relatively un tu tored subjects to identify the objects represented in pictures. No one has p roposed that the ability to use these codes is innate. H o w -

INSTRUCTIONAL RESEARCH ON VISUAL LITERACY : 31

ever, very young children, people in cultures lacking pictorial materials, and even chimpanzees, can identify familiar objects in pictures at first sight. Further evidence of the primi- tive nature of this ability is provided by the classic study by Hochberg and Brooks (1962), who reared their son in a pic- tureless environment until he was 19 months old. When he was then presented with pictures for the first time, he could identify familiar objects in simple and complex line drawings and in photographs. 3

Certain other codes used in a wide range of pictorial media require somewhat more skill. Researchers have found that some subjects, while able to identify the objects in a picture, cannot tell how the objects are arrayed in the third dimension (see Hagen, 1974). The ability to interpret depth information available in codes such as linear perspective and overlap is apparently more difficult to acquire than the ability to sepa- rate figure from ground. A similar type of code is the use of disequilibrium to suggest motion in still pictures. For example, the meaning intended by drawing the pins on a bowling alley at various oblique angles will not be decoded by many young children. Sources of hypotheses for other codes of this kind which may apply to "pictures in general" can be found in the literature of the psychology of visual perception and art (e.g., Arnheim, 1969; Gibson, 1971; Goodman, 1976; Ken- nedy, 1974).

Moving on down the continuum, codes that are characteristic of a group of related media can be identified. The ability to use such codes is what some people call "media literacy." This ability not only involves being able to interpret the information available in the stimulus but also may involve the development of new mental skills.

Underlying the use of media . . . are three assumptions: (a) symbolic codes serve for both communicational and representational purposes; (b) external coding systems that serve for communication purposes can be incorporated or internalized to serve in a representational capacity; and (c) the codes, once internalized, can be schematized (i.e., detached from their original contexts) and thus serve as schemes of thought. (Salomon, 1974, p. 402)

3Note that to interpret a picture a person must have knowledge of the referent. For example, a picture of a telephone will have little meaning to an African bushman.


IMPLICATIONS

FOR RESEARCH O N TEACHING

VISUAL LITERACY

Accordingly, Salomon and Cohen (1977) demonstrated that different television formats involving editing techniques and the use of the zoom lens differentially affect the mental skills called into play. In a similar vein, Worth and Adair (1972) observed that Navajos who were taught how to use film production equipment but were not taught conventional "film grammar" developed film codes that mirror mental processes characteristic of their own culture. Thus, some codes seem to arise from the technology of the media involved (e.g., the use of high-speed, slow-motion, and time-lapse photography to depict the mental manipulation of time) while others apparently arise more from the desire to represent mental operations technically (e.g., the use of split-screen and cross- cutting to depict simultaneity).

Toward the bottom of the continuum there is a range of codes that tend to have the status of pictorial conventions and to be more content specific. The range of subject matter information these subordinate codes can carry is more limited. Some examples of pictorial cues involved in such codes are shown in Figure 3. The dotted lines from the baker's eyes to the empty bread tray is a convention to indicate a character's line of gaze. The wavy lines above the loaves of bread are cues for heat rising, and the straight lines radiating from the light bulb indicate that the light is on. The trailing edge lines behind the running figure are one type of cue used to indicate motion. Several researchers (e.g., Friedman & Stevenson, 1975) have shown that young children are unable to interpret these conventional, implied motion cues. Other kinds of pictorial conventions are used in other media such as maps and architectural drawings.

Of course Figure 3 also employs more superordinate codes such as linear perspective and disequilibrium (the spoon is perceived not as floating in midair, but as falling from the baker's hand). Digital signs appear as well, as in the cases of the words "Bob's Bakery" and the octagonal traffic sign which we have learned means "stop." The no smoking sign above the door is an example of combining an iconic sign (the cigarette) and a digital sign (the slash, which has arbitrarily been chosen to signify negation). The degree of iconicity between the structural appearance of a pictorial cue and its corresponding mental representation appears to differ for different iconic codes. The structures of cues used in superordinate codes seem to be rather "natu-

INSTRUCTIONAL RESEARCH O N VISUAL LITERACY : 33

FIGURE 3

Pictorial Cues: Some Examples

f

-�9

ralistically imperative." For example, the forms that depth cues can take are largely determined by the rules that govern our perception of the natural environment. On the other hand, pictorial conventions found toward the bottom of the continuum may be relatively arbitrary in appearance. Such conventions are not totally arbitrary; the trailing edge lines are similar to the perceptual blurring that is experienced when observing a fast moving object, the wavy "heat rising" lines are similar to what one sees looking down a highway on a hot day, and so forth. But in many cases other similar cues could, and do, function equally well. Also, as previously mentioned, subordinate codes tend to be more situation specific and information specific.

Assuming the validity of these observations, some hypotheses about techniques for teaching visual literacy can be offered. It seems reasonable that subordinate codes could be


successfully taught by paired-associate paradigms used, for example, in foreign language instruction. Given that the ap- propriate response is in the learner's repertoire, the task is to learn to associate this response with the correct pictorial stimulus. When the situations and range of information the code deals with are limited, relatively few examples will pro- vide the learner with a complete understanding of the code.

A complete understanding of superordinate codes, such as linear perspective, is more difficult to acquire. While adults rarely have trouble interpreting linear perspective, a more difficult interpretive task-- ident i fying failures to use the code properly in a poorly made drawing, for example--wil l baffle most adults. More complex teaching methods are required, including, perhaps, instruction in drawing in one-, two-, and three-point perspective. Thus, while the ability to interpret a code is not usually dependent upon being able to produce it (externalize it), production may be an excellent avenue to a richer understanding. 4 This may be particularly true for codes that are not well understood by anyone. For example, the interpretation of Figure 3 requires much more than an understanding of the pictorial conventions used. Re- lating the elements in the picture to reconstruct the theme the artist had in mind entails the use of codes that are difficult to analyze and teach. Asking learners to produce their own meaningful pictures (drawing, arranging cutouts of figures, etc.) may facilitate learning such codes. In such cases the critical aspect of attaining interpretive skills may be the acquisition of the mental operation itself. In any event, such questions seem apt targets for research.

Who is in need of instruction in visual literacy? Most of the subjects of research cited in this paper are either young children or adults in developing areas of the world. While such subjects do need instruction, the fact that most adults in our culture are unskilled in extracting the information available in pictures must not be overlooked. Jahoda, Cheyne, Dere- gowski, Sinha, and Collingbourne (1976), in their study of the effects of text illustrations, found that while pictures helped adults learn information that was also covered in the text,

4Also, production is usually a more reliable criterion for testing understanding. For example, testing the acquisition of film editing concepts such as "jump cut" by asking for verbal definitions can result in inflated estimates of understanding when compared with students' abilities to produce examples and nonexamples of concepts.

INSTRUCTIONAL RESEARCH O N VISUAL L1TERACY : 35

Basic Research Questions

information available in the pictures that was not also covered in the text was not learned. It is likely that people's responses to pictures seldom go beyond a simple labeling response ac- companied by a fleeting affective reaction. In other words, people do not "s tudy" pictures, and usually learn very little from them, although they are influenced by them (e.g., ad- vertising). In summary, it has been suggested that instructional research in visual literacy focus on the study of the symbolic codes of pictorial media. The key question for researchers is: What pictorial cues serve to evoke what mental operations? Or, put differently: What mental operations can be represented picto- rially? Further questions involve the identification of effective strategies for teaching particular codes and the identification of the types of subject matter information particular codes can best carry. I recommend that such research be conducted within the framework of the iconic mode. While the importance of verbal processes is not denied, the transactions between pictorial stimuli and mental imagery are most fundamental to the study of visual literacy.

REFERENCES Arnheim, R. Visual thinking. Berkeley: University of California Press, 1969.

Arnheim, R. Virtues and vices of the visual media. In D. R. Olson (Ed.), Media and symbols: The forms of expression, communication, and education (National Society for the Study of Education Yearbook, Part 1). Chicago: National Society for the Study of Education, 1974.

Debes, J. L. The loom of visual literacy--An overview. In C. M. Williams & J. L. Debes (Eds.), Proceedings of thefirst national conference on visual literacy. New York: Pitman, 1970.

Fleming, M. L. The picture in your mind. AV Communication Review, 1977, 25, 43-62.

Friedman, S. L., & Stevenson, M. B. Developmental changes in the understanding of implied motion in two-dimensional pictures. Child Development, 1975, 46, 773-778.

Gibson, J. J. The information available in pictures. Leonardo, 1971, 4, 27-35.

Goodman, N. Languages of art (2nd ed.). Indianapolis: Bobbs-Merrill, 1976.

Hagen, M. A. Picture perception: Toward a theoretical model. Psy- chological Bulletin, 1974, 81, 471-497.

Hochberg, J. E., & Brooks, V. Pictorial perception as an unlearned ability: A study of one child's performance. American Journal of Psychology, 1962, 85, 624-628.


Jahoda, G., Cheyne, W. M., Deregowski, J. B., Sinha, D., & Col- lingbourne, R. Utilization of pictorial information in classroom learning: A cross cultural study. AV Communication Review, 1976, 24, 295-315.

Just, M. A., & Carpenter, P. A. Eye fixations and cognitive processes. Cognitive Psychology, 1976, 8, 441-480.

Kennedy, J. M. The psychology of picture perception. San Francisco: Jossey-Bass, 1974.

Knowlton, J. Q. On the definition of "picture." AV Communication Review, 1966, 14, 157-183.

Kosslyn, S. M., & Pomerantz, J. R. Imagery, propositions, and the form of internal representations. Cognitive Psychology, 1977, 9, 52-76.

Paivio, A. Imagery and verbal processes. New York: Holt, Rinehart and Winston, 1971.

Pylyshyn, Z. W. What the mind's eye tells the mind's brain: A cri- tique of mental imagery. Pschological Bulletin, 1973, 80, 1-24.

Salomon, G. What is learned and how it is taught: The interaction between media, message, task, and learner. In D. R. Olson (Ed.), Media and symbols: The forms of expression, communication, and education (National Society for the Study of Education Yearbook, Part I). Chicago: National Society for the Study of Education, 1974.

Salomon, G., & Cohen, A. A. Television formats, mastery of mental skills, and the acquisition of knowledge. Journal of Educational Psychology, 1977, 69, 612-619.

Shepard, R. N. The externalization of mental images. Paper presented at the conference, A Retrospect and Prospect of Visual Learning, Thinking and Communication, sponsored by The Visual Schol- ars' Program, Iowa City, October 1976.

Shepard, R. N., & Chipman, S. Second-order isomorphism of internal representations: Shapes of states. Cognitive Psychology, 1970, 1, 1-17.

Shepard, R. N., & Metzler, J. Mental rotation of three-dimensional objects. Science, 1971, 171, 701-703.

Worth, S., & Adiar, J. Through Navajo eyes: An exploration in anthropology and film communication. Bloomington: Indiana Univer- sity Press, 1972.

Documents

A prospectus for instructional research on visual literacy