Upload
berny
View
58
Download
0
Embed Size (px)
DESCRIPTION
Languages, Images and Words. Week 7 :CCT370 – Introduction to Computer Visualization. Addresses relationships between visual information and verbal or textual information Dual coding theory When should we use a visual display? What is a visual language? - PowerPoint PPT Presentation
Citation preview
Languages, Images and Words
Week 7 :CCT370 – Introduction to Computer Visualization
Introduction: Pictures and Words
Addresses relationships between visual information and verbal or textual information
Dual coding theory
When should we use a visual display?
What is a visual language? And does it make sense to use one to program a computer
How to integrate images and words
A Memory Demonstration
Remembering words (and lots of things) is not easy
Given a is a list of 10 words
Try to remember them
Method of Loci - a Mnemonic Remembering things, especially long sequences, not a new problem
Speech, tasks to do, names, …
Dating back to Greek times, have been various mnemonic devices
Method of loci, or places Uses places and encoding of items E.g., take a walk through your apartment, or across campus, or your parents’ house – some well
known place A “memory palace”
In Renaissance, mol widely used, cathedrals served nicely Identify several points
E.g., 10 for the demonstration When a sequence is given, place each element at a (physical) point on the path
The more “memorable”, or bizarre, the image the better To retrieve the elements, walk through the physical place
Memory demo again, but will do it using method of loci
Decide on a blueprint for your palace. The larger or more detailed the real place,
the more information you can store in the corresponding mental space.
Define a route. If you will need to remember things in a
certain order, it is essential that you follow a specific route through your palace, both in the real world and in you mind.
Identify specific storage locations in your palace or along your route.
Identify as many locations as you think you will need.
Walk through your structure or along your route and really observe it.
Memorize your memory palace.
Place things to be remembered in your palace.
Put a manageable amount of information in each place.
Use symbols. Generally, all you need to store in each
location is something that will jog your memory, something that will lead you to the actual idea you’re trying to remember.
“How to Build a Memory Palace”http://www.wikihow.com/Build-a-Memory-Palace
• Be creative. – Generally, images will be more
memorable if they are absurd (out of the ordinary
• Stock your palace with other mnemonics.
– “Every Good Boy Deserves Fudge," which would in turn allow you to recall the order of notes on the lines in treble clef (EGBDF).
• Explore your palace. – Go through it and look at them.
• Use your palace. – just follow your route in order as
you do so.
• Build new palaces. – If you need only commit things
to memory for a short time. Just replace the existing contents with new ones
– If you need to remember the contents of your palace for a long time, you can keep that palace as it is and create new ones in which to store other information as needed
Memory Demo, Again Same song – new technique
Coding Words and Images: Two Systems
Bertin, Semiology of Graphics (1983) Two distinct sign systems
Associated with auditory information processing Includes mathematical symbols, natural language, music
Based on visual information processing Includes graphics, abstract and figurative imagery
Pavio (1987), dual coding theory Two different types of information in working memory
Imagens – mental representation of visual information Objects, natural groupings of objects, whole parts of objects (e.g., arm), spatial
information about layout/arrangement Logogens – mental representation of language information
Basic information pertaining to language, but not sounds of words Processed by a set of functional subsystems that provide support for reading
and writing, understanding and producing speech, logical thought Not necessarily tied to speech, e.g., deaf and Braille and sign language
Pavio’s Dual Coding Theory, 1 Account of memory (with implications for perception)
Refines/extends basic approach of 3-stage model of human memory
Sharpens ideas about object recognition
“Makes sense” – split between visual and verbal processing Long known are different neural processing centers for:
Verbal information (speech areas of temporal cortex) Visual information (visual cortex)
Core of rationale for use of visual representations Especially to facilitate learning
E.g., concrete mnemonic devices Greek orators … Walk through your house and put …
Primary rationale for “multimedia”! Provides account of why it works
For our purposes, informs use of visual representation With “words”, in larger context of diagrams, etc. Sharpening up relation of “visual” with “semantic, …”
Processing: Visual-spatial information Visual text Acoustic verbal stimuli
Pavio’s Dual Coding Theory, 2 Processing:
Visual-spatial information Enters through visual system
Fed into association structures in nonverbal imagen system
Visual text Processed,
But then fed into association structures of logogens
Acoustic verbal stimuli Processed in auditory system
Then fed into logogen system
Logogens and imagens can be strongly interlinked E.g., word “cat” and language-
based concepts related to cats will be linked to visual information related to the appearance of cats
Method of loci, multimedia, etc.
•Imagens – mental representation of visual informationObjects, natural groupings of objects, whole parts of objects (e.g., arm), spatial information about layout/arrangement
•Logogens – mental representation of language informationBasic information pertaining to language, but not sounds of wordsProcessed by a set of functional subsystems that provide support for reading and writing, understanding and producing speech, logical thoughtNot necessarily tied to speech, e.g., deaf and Braille and sign language
“Thinking” Visually Usually associate processes of “thinking”, e.g.,
greater than, follows from, “logic”, etc., with “verbal” (logogen based system)
More recently, such operations as part of the “visual” (imagen based system) are finding evidence E.g., Subjects report using imagery when compare
sizes of light bulb and tennis ball, color (green) of pea and Christmas tree
Positive emission tomography (PET) evidence Visual processing centers active when imagery invoked Buy, when mentally change size and position of
imagined object, different visual areas of brain activated
Imagery alone vs. operations on (processing, thinking with) imagery When see a cow and imagine a cow same neural
pathways (in part) excited In accord with accounts of object recognition in
which object is “recognized” through interaction of stimuli (bottom up) and memory (top down) See figure at right
Visual memory traces stored as part of processing Hence, recognition is easier than recall
Matching with something stored (which in part drives recognition)
Vs. reconstructing all pieces from input of event alone
The Nature of Language(s) Noam Chomsky:
Innate deep structures (meaning) Surface structure (syntax, form)
That nature of natural (everyday, spoken) language is quite similar to formal description of languages appropriate to describe all languages (natural and formal) is among larger advances of 20th century Universality of human language And, e.g., Chomsky hierarchy of (deep) interest to
computer scientists
Critical period for language development
But being verbal is not essential to language development
Sign languages for the deaf are the most perfect examples of visual language
What is Language? Language provides:
Description Communication of intention Ability to communicate
procedures and sequences of operations – including logic if, but, causes, do a then b
then c
Sign Languages Arose spontaneously Are not related to verbal
languages Have syntax Become more abstract To be fluent in visual (or any)
language we should be trained from early in life
Again, Visual Languages (or Not)
Recall, some fundamental questions addressed in first of class Sensory and arbitrary symbols, etc. Visual language
Consider that hieroglyphs gave way to more abstract symbols Why turn back the clock?
Can there be a true visual language? Yes, but not for most of us! A critical (developmental) period
ontogenetic Consider verbal language Abstraction, logic
(if, while, perhaps) Based on speech Sign languages are true visual languages
Visual and Verbal Pseudo-code
Ware argues that visual languages in fact not good for expressing sequential operations common to computer programming,
Sequence below: Get a line of text Change characters to all upper case Write line to output file While there is more input
Better expressed by “verbal psuedo-code”:Repeat
Get a line of textChange characters to all upper caseWrite line to output file
Until there is more input
Flow charts were wrong Probably just wrong paradigm
But, billions of dollars spent Cause inappropriate/unnatural focus on detail
Visual programming languages have history of failure
Still, Certainly Uses for Visual Represt.
Gives rapid recognition and pattern finding (again) Has distinct advantages over text (linear, serial) for
conveying some kinds of information Consider the text below:
Jane is Jim’s boss Jim is Joe’s boss Anne works for Jane Mark works for Jim Anne is Mary’s boss Anne is Mike’s boss
And it’s visual representation
When to Use Static Images vs. Words
In general (rules of thumb) Images are better than words for:
Spatial structures Location Detail
Words better for: Procedural information Logical condition Abstract verbal concepts
Images best for showing structural relationships Links between entities and groups of entities
E.g., bus routes shown as graphical representation led to better performance in trip planning than with tables
Visual information generally remembered better than verbal, but not for abstract images Visual information need be meaningful and capable of incorporation into a cognitive framework for this advantage Image memory can’t be so relied on if information is new and out of context
Images best for providing detail and appearance Amount of detail extracted (and remembered) depends on time to study
Recall, silhoettes first, so line drawing best for rapid extraction
Information that specifies conditions under which something should or should not be done is better provided using text or spoken language
Gestures, and Linking Images and Words
Deixis and the deictic gesture Gesture that links subject of a spoken sentence with a visual reference Can be a glance or a nod Pre-speech Shown to disambiguate verbal communications
Why the mouse is so powerful … Other kinds of gestures
Beat gestures for emphasis Verb gestures showing how to do something McNeil Hand and mind
Issues in shared environments Speech + Pointer + Visuals – most important components Subtle ways of directing attention also important in meeting dynamics.
Linking images and words Deixis Pointing is an elementary speech act. Pointing links images and words Put that (points) there (points) Subject verb predicate
Attaching Words to Images Turns out the use of words and images together work just
fine Or even quite well
“Association” (grouping) can be explained by Gestalt principles
Examples of Integrated Pictures and Words
Examples
Another Ex. of Integrated Pictures and Words
More examples
Yet Another Ex. of Integrated Pics. and Words
Examples of visual languages As noted, limited
success of visual languages, especially for programming
Still, some examples Sanscrit Petri-nets Khoros
Sanscrit To program:
Count from 1 to 3 for i = 1 to 3 do
Petri Net Language Petri nets are
stochastic timed attributed (tokens
on nodes, transitions)
Khoros Combine operations
Conclusions Interesting, and not fully understood interaction
between the two processing subsystems “Visual”, images, imagens “Verbal”, auditory, logogens
Though certainly there can be “visual languages”, finding useful applications has been elusive
Next class Topic: Thinking Visualizations Readings:
Ware, Chapter 11