Theories of Vision: a swift overview Learning-based Methods in Vision A. Efros, CMU, Spring 2012 Most slides from Steve Palmer

Theories of Vision: a swift overview

Learning-based Methods in VisionA. Efros, CMU, Spring 2012

Most slides from Steve Palmer

What is a Theory?What is a Theory?

Theory An integrated set of related hypotheses about underlying mechanisms or principles that organize, explain, and predict facts about the phenomena of a given domain.

© Stephen E. Palmer, 2002

What is a Theory?What is a Theory?

Three important functions of a theory:Integrate “old” factsPredict “new” factsLead to “understanding”


Three criteria for evaluating a theory:Logical consistencyEmpirical adequacyParsimony (“Occam’s razor”)

Categorization of Theories of Vision

“Why do things looks as they do?”-- Kurt Koffka (1935)

Nativism vs. Empiricism• “Because we were born (evolved) to see them that way” vs.• “Because we have learned to see them that way”

Atomisn vs. Holism• “because of the way each pixel appears” vs.• “because of the way the entire scene appears”

Organism vs. Environment• “Because we are the way we are” vs.• “Because the world is the way it is”

StructuralismStructuralism


THEORY NATIVISM vs.EMPIRICISM

ATOMISM vs. HOLISM

ORGANISM vs.ENVIRONMENT

PRINCIPAL ANALOGY METHOD

EcologicalOptics

Structuralism

Gestaltism

Constructivism



Structuralism: Perception results from the association of basic sensory atoms in memory via repeated, prior joint occurrences.

Derived from philosophy of British Empiricists (e.g., Locke, Berkeley, Hume, and Mills).

Proposed by Wilhelm Wundt, the father of modern Psychology.



Sensory Atoms

Retinal mosaic

Greennessat (x3,y3)

Yellownessat (x2,y2)

Rednessat (x1,y1)



Perceptual Complexes

Retinal mosaic




Retinal mosaic

Red appleat (x0,y0)




Retinal mosaic

Red appleat (x0,y0) Association



Chemical Analogy

Perceptions are made of basic sensory experiences just as molecules are made of basic atoms.




ATOMISM vs. HOLISM



EcologicalOptics

Structuralism

Gestaltism

Constructivism

Empiricism Atomism Organism ChemistryTrained

Introspection

GestaltismGestaltism


Gestaltism: Perception results from the interactionbetween the intrinsic structure of the stimulusand the intrinsic structure of the brain.

MaxWertheimer

WolfgangKöhler

KurtKoffka



The Gestalt movement in perceptual theory was primarily a reaction against Structuralism:

Successful in arguing against Structuralism,but less successful in promoting its own theoretical agenda.

Rejected atomismRejected empiricismRejected associationism


Holism: The whole is different from the sum of its parts.

Principles of Gestalt Theory

Prägnanz: The percept will be as “good” as the prevailing conditions allow, I.e. simplest explanation

Nativism:Not a total rejection of learning, but rejection of Its primacy.



Prägnanz: the percept will be as “good” as the prevailing conditions allow

What is this?

?square &

circle?

?square &pacman?

?squigit &pacman?

etc….



Holism: The whole is different from the sum of its parts.

Emergent properties:Features of a configurationthat are not features ofits components, e.g.:

• length• orientation• curvature• closure• connectedness


The problem of grouping: “I stand at the window and see a house, trees, sky.”Theoretically, I might say that there were 327 brightnesses and nuances of color. Do I have “327”? No. I have sky,house, and trees. (Wertheimer, 1923)

The Problem of OrganizationThe Problem of Organization


Wertheimer’s “laws” of grouping

Rows

Perceptual GroupingPerceptual Grouping

Columns


14.20

Proximity

Rows Columns



14.21

Color Similarity

Rows Columns



14.22

Size Similarity

Rows Columns



14.23

Orientation Similarity

Rows Columns



14.24

Similarity of texture

Rows


Columns


14.26

Common Fate

Columns



14.27

Good Continuation

ColumnsRows



14.28

Closure

ColumnsRows



14.29

Common Region

Rows


Columns


14.30

Element Connectedness

Rows


Columns


14.32

Past Experience




Field Theoretic Analogy

Field ofa singlepositivecharge

Field ofa pair ofpositivecharges

Field of apositive &a negative

charge

Repulsion Attraction




ATOMISM vs. HOLISM



EcologicalOptics

Structuralism

Gestaltism

Constructivism

Nativism Holism Organism EM FieldsNaive

Introspection

Trained Introspection

Empiricism Atomism Organism Chemistry

Ecological OpticsEcological Optics


Ecological Optics: Perception is the direct apprehension of the visible environment by extracting invariants in the dynamic ambient optic array and the affordances of objects.

James J. Gibson: Ask notwhat’s inside your head, butwhat your head’s inside of!(Ecological optics is a theory of stimulus structure.)



Direct Perception: Perception is not mediatedby processes like unconscious thinking

Perception of the 3-D world is NOT ambiguous for an actively exploring organism.



Texture Gradients are gradual changes in the size & shape of texture elements in an image when a plane recedes in depth.

Tilefloor

Wheatfield

Geometric surface



Motion Gradients are gradual changes in the speed and direction of elements in optic flow within the dynamic ambient optic array.

Bird flyingPlane landing




ATOMISM vs. HOLISM



EcologicalOptics

Structuralism

Gestaltism

Constructivism



PhysicalField Theory

NaiveIntrospection

Nativism Holism Organism

Nativism Holism EnvironmentMechanicalResonance

EcologicalAnalysis

ConstructivismConstructivism


Constructivism: Perception is the result ofunconscious inferences about the scene mostlikely to have caused the retinal image or event.

Hermann von Helmholtzoriginated the idea ofunconscious inference andthe likelihood principle.



Unconscious Inference: the process of recovering environmental information by logically combiningretinal information with heuristic assumptions.

Tilted room illusion:If you assume that thewalls and floor of the room are vertical andhorizontal, then youmust be tilted —and youfeel that way!



Tilted room illusion

A. Actual Situation

B. Perceived Situation



Prägnanz vs. LikelihoodWhat governs what we see: goodness or probability?

What is this?

? ? ?square &

circle?square &pacman?

squigit &pacman?

etc….




ATOMISM vs. HOLISM



EcologicalOptics

Structuralism

Gestaltism

Constructivism



PhysicalField Theory

NaiveIntrospection

Nativism Holism Organism

(both) (both) (both)Logical

InferenceLikelihoodAnalysis

Mechanical Resonance

EcologicalAnalysis

Nativism Holism Environment


So which theory is correct?

Probably none of them!

Or maybe all of them, to some degree!

It’s good to keep them in mind, whendesigning your algorithms!

“You can’t play 20-questionswith nature and win!”

(Allen Newell, 1973)

Four Stages of Visual PerceptionFour Stages of Visual Perception


Image- BasedProcessing

Surface- BasedProcessing

Object-Based

Processing

Category- BasedProcessing

Light

Vision

Audition

STM

LTM

Motor

Sound

LightMove-ment

Odor (etc.)

Ceramiccup on a table

David Marr, 1982



The Retinal Image

An Image (blowup) Receptor Output



Image-basedRepresentation

Primal Sketch(Marr)

An Image

(Line Drawing)

RetinalImage

Image-based

processes

EdgesLinesBlobsetc.



Surface-basedRepresentation

Primal Sketch 2.5-D Sketch

Image-basedRepresentation

Surface-based

processes

StereoShadingMotion

etc.

Koenderink’s trick



Object-basedRepresentation

Object-based

processes

GroupingParsing

Completionetc.

Surface-basedRepresentation

2.5-D Sketch Volumetric Sketch

Geons(Biederman '87)



Category-basedRepresentation

Category-based

processes

Pattern-Recognition

Spatial-description

Object-basedRepresentation

Volumetric Sketch Basic-level Category

Category: cup

Color: light-gray

Size: 6”

Location: table

We likely throw away a lot

line drawings are universal

However, things are not so simple…

● Problems with feed-forward model of processing…

two-tone images

hair (not shadow!)

inferred external contours

“attached shadow” contour

“cast shadow” contour

Finding 3D structure in two-tone images requires distinguishing cast shadows, attached shadows, and areas of low reflectivity

The images do not contain this information a priori (at low level)

Cavanagh's argument

Marr's model (circa 1980) Cavanagh’s Model (circa 1990s)

Feedforward vs. feedback models

stimulusstimulus

2D shape

memory

3D shape

2½D sketch

Object

3D model

feedback

basic recognition with 2D primitives

reconstruction of shape from image features

object recognition by matching 3D models

primal sketch

A Classical View of Vision

Grouping /Segmentation

Figure/GroundOrganization

Object and Scene Recognition

pixels, features, edges, etc.Low-level

Mid-level

High-level

A Contemporary View of Vision

Figure/GroundOrganization

Grouping /Segmentation

Object and Scene Recognition

pixels, features, edges, etc.Low-level

Mid-level

High-level

But where we draw this line?

Documents

Theories of Vision: a swift overview Learning-based Methods in Vision A. Efros, CMU, Spring 2012 Most slides from Steve Palmer