EE1411

Perception and attentionPerception and attention

Janusz A. Starzyk

Computational IntelligenceComputational Intelligence

Based on a course taught by Prof. Randall O'Reilly University of ColoradoProf. Włodzisław DuchUniwersytet Mikołaja Kopernika andProf. Oliver University of ConnecticutSchool of Medicine

EE1412

MotivationMotivationPerception is comparatively the easiest to understand although for many specific questions there are no clear answers.

General questions:

Why does the primary visual cortex react to oriented edges?

Why does the visual system separate information into the dorsal stream, connected to motion and representation of object locations, and the ventral stream, connected to object recognition?

Why does damage to the parietal cortex lead to spatial orientation and attention disorders?

In what way do we recognize objects in different places, orientations, distances, with different projections of the image onto the retina?

EE1413

VisionVision

The hierarchical organization begins in the retina, passes through the lateral geniculate nucleus (LGN - part of the thalamus), reaching the primary visual cortex V1, from where it's distributed further.

EE1414

Visual systemVisual systemSight in different types of animals is realized in many ways: a snail has light-sensitive cells without lenses, insects have a complex eye and10-30,000 hexagonal facets, mammals have an eye with a retina and a lens, people have around 120M receptors.

EE1415

Visual systemVisual system

EE1416

Visual systemVisual systemV1 extends rostrally almost to the lunate sulcus and posterolaterally almost to

the inferior occipital sulcus; the V1/V2 border is met before either sulci.

There are three basic types of neurons in the primate V1 (Fig 12):Spiny pyramidal cells (excitatory) Spiny stellate cells (excitatory) Smooth or sparsely spinous interneurons (almost all are GABAergic).

Pyramidal and stellate cells

Local axon, double bouquet, basket, chandelier, bitufted, neurogliaform cells

EE1417

Visual pathwaysVisual pathwaysVisual pathways: retina => lateral geniculate nucleus (LGN) of the thalamus=> visual radiation => area of the primary cortex V1 => higher levels of the visual system => associative and multimodal areas.

V1 cells are organized in ocular dominance columns and orientation columns, retinoscopic. Simple layer 4 cells react to bands with a specific slant, contrasting edges, stimulus from one eye. A substantial part of the central V1 area reacts to signals from fovea, where the density of receptors is the greatest.

EE1418

depth

direction

“where” pathway

“what” pathway

orientation

shape

color

RecognizedObject ready for perception

“Where" = large-celled pathway, heading for the parietal lobe.

"What"= small-celled pathway heading for the temporal lobe (IT).

Two streams whereTwo streams where?/what??/what?

EE1419

Two streams where?/what?Two streams where?/what?

Milner and Goodale (1995): visual pathways don't so much determine where and what, as much as they enable action and perception.

There is also the old limbic pathway, enabling rapid action in dangerous situations (after which follows a wave of fear).

What? - temporal lobe

Where? - parietal lobe

EE14110

Two streamsTwo streamsUngerleider and Mishkin (1982): there exist two notably divided pathways for processing visual information, running from the eye. Large-grained PA retina cells, 3 types of photoreceptive cones, large receptive fields, rapidly-conducting axons, activation for light in a wide band. Small-grained PB cells, 1 or 2 types of photoreceptive cones, small receptive fields, slowly conducting axons, recognize color oppositions.

Large-celled pathway: runs to two large-celled LGN layers, it's characterized by a low spatial resolution, high sensitivity to contrast, rapid signal transfer, without information about color. The small-celled pathway has 4 small-grained layers in the LGN, high spatial resolution, color, slower information transfer, low sensitivity to contrast.

EE14111

RetinaRetina The retina is not a passive camera registering images. Crucial rule: enhancing contrasts underlining changes in space and

time, strengthening edges, uniformly lit areas are less important. Photoreceptors in rods and cones, 3-layer network, ganglion cells =>LGN.

Receptive fields: areas, which stimulate a given cell. The combination of signals in the retina gives center-surround receptive fields (on-center) and vice versa, detects edges. Each individual field of cells can be modeled as a Gaussian model, so these fields are obtained as a difference of Gaussians (DOG).

EE14112

Lateral geniculate nucleusLateral geniculate nucleus Signal compression – partly already done in the retina. Different types of information find their way to different LGN layers. Intermediate station – all sensory signals (except olfactory) go

through different nuclei of the thalamus. Dynamic information processing: steering attention and fast large-

celled pathway reacting to motion. Retroactive projections V1=>LGN are an order of magnitude more

numerous than LGN=>V1 (role - prediction).

The competitive dynamic selects signals from the visual field, especially involving motion.

EE14113

LGN of the ThalamusLGN of the Thalamus

Parvocellular layers 3-6

Magnocellular layers 1& 2

EE14114

Edge detectorsEdge detectorsContrasting signals points from the LGN are organized by the V1 cortex into edge detectors oriented at a specific angle.

Simple V1 cells combine into edge detectors, enabling the determination of shapes, other cells react to color and texture.

Properties of edge detectors: different orientation; high frequency = fast changes, narrow bands;

low frequency = gentle changes, wide bands;

polarization = dark-light or vice-versa, dark-light-dark or vice-versa.

EE14115

Topography of the cortexTopography of the cortexSpecific construction, partly genetically determined, partly develops

thanks to stimulation, retinotopic organization like in the LGN.

Different types of edge and texture detectors are topographically packed

in the V1 cortex into hypercolumns, containing separate signals from the

left and right eyes (3D vision, not in all mammals).

Blob region: signals of color and partly of shape, low frequencies => V4.

Interblob region: edge detectors, every 10o, high frequencies.

Hypercolumn ~1mm2, ocular dominance columns in layer 4.

EE14116

Representation in the V1 cortexRepresentation in the V1 cortex

Oriented edge detectors can be created by correlational Hebbian learning

based on natural scenes.

What happens with information about color, texture, motion?

EE14117

Retinoscopic maps in V1Retinoscopic maps in V1

The spatial position of the ganglion cells within the retina is preserved by

the spatial organization of the neurons within the LGN layers. The

posterior LGN contains neurons whose receptive field are near the fovea.

EE14118

Area V1: The Primary Visual CortexArea V1: The Primary Visual Cortex

V1 is made up of 6 layers (no relation to 6 layers in LGN).

LGN sends axons to layer IV of V1. M and P cells are

separate. Right and Left eye

are separate.

EE14119

Retinal ganglion cellsRetinal ganglion cells There are two types of ganglion cells in the retina:

Large magnocellular ganglion cells, or M cells, carry information about:

– Movement– Location– depth perception.

Smaller parvocellular ganglion cells, or P cells, transmit signals that pertain to:

– Colour– Form– texture of objects in the visual field.

EE14120

The geniculostriate pathway The geniculostriate pathway The M cells send their

information to layers 1 & 2. The P cells send their

information to layers 3-6. So, layers 3-6 are involved

in processing information concerning fine detail and color.

Layers 1 & 2 process information concerning movement.

EE14121

Dorsal pathwayDorsal pathway

Large-celled pathway: from the occipital lobe through the dorsal pathway to the parietal cortex. Arrives at the 4B layer in V1, from here to the thick dark stripes of the V2 region, analyzes information about object motion.

In V1, layer 4B => V5, localization in the field of vision, motion.

V5 stimulates the parietal lobe, PPC (posterior parietal cortex), regions 7 and 5; this enables spacial orientation, depth and motion perception(eye orientation).

EE14122

Ventral pathwayVentral pathway

Small-celled pathway: the ventral pathway, to the inferior temporal cortex.

V1 => V2 interblob region, reacts to line orientation, gives a large visual acuity, without color. V1 => V3 blob region, reacts to shapes, reaction to color in the neurons in the dark stripes of V3. V2 => V4, main area of color analysis, information arrives at the inferior temporal cortex (IT). The IT area in the inferior temporal lobe has neurons which react to complex objects.

EE14123

Model v1rf.proj.gz, Chapt. 8Model v1rf.proj.gz, Chapt. 8How do receptive fields form? Where do these V1 properties come from?

Description of the project in Chapt. 8.3.2. Natural shapes and textures

lead to specific receptive fields: from this come reactions to edges.

Inputs: 12x12, signals from LGN

cells: on- and off-center.

Input images: randomly chosen

from a natural 512x512 scene.

Hidden layer 14x14; connections:

coincidental with the input,

excitatory with the surroundings.

EE14124

Model propertiesModel propertiesThe V1 cortex receives from the LGN an on/off signal with heightened

contrast, input to V1 through layer 4, processing in this model responds to

overlapping processes mainly in layers 2 and 3.

The model includes one hypercolumn, analyzing a small sector of the image

from images of landscapes and plants => all elements see the same thing.

Properties: spherical geometry, i.e. top = bottom, left = right; independent inputs for on/off cells, in accordance with biology;

strong and widespread excitatory horizontal connections – like in SOM; kWTA leaves ~10% active neurons.

Contrast for weights is small ~1, because these aren't decision-making neurons, thresholds are large (~2) to force sparse representations, strong correlations.

Noise helps in avoiding weak solutions.

EE14125

Exercises fromExercises from v1rf v1rf

Check the structure, connection weights (r.wt): strong activations within

the hidden layer, random connections with on/off inputs.

LoadEnv to load the 512x512 image - for the training 10 images were

used, here is one random one, processed into on/off points.

StepTrain – observe the oscillation of learning for phases – and + Complementarity of on/off: stronger "on" activation for images which are brighter in the middle than on the edges, dark = extra "off" activation.

Question: what can we expect if horizontal connections will dominate? Check your predictions, temporarily changing lat_wt_scale 0.04 => 0.2.

LoadNet to load the trained network, after 100,000 presentations of images and several days of calculations...

EE14126

Receptive fieldsReceptive fieldsHow do receptive fields form? Where do these V1 properties come from?

View, PROBE_ENV shows 4 different probe stimuli, StepProbe will show

activation of hidden units.

Check the activation r.wt, change the color scale so we can better see the field orientation, check several hidden elements, bi- and tri-polar fields of both types.

Load all: View, RFIELDSactivation on=red, off=blue. Orientation, position, size, polarity are 4 different traits of receptive fields.

Radial orientation changes (pinwheels), singular points.