A Mechanism for Learning, Attention A Mechanism for Learning, Attention Switching, and CognitionSwitching, and Cognition

School of Electrical Engineering and Computer Science, Ohio University, USAhttp://people.ohio.edu/starzykj

Dagstuhl Seminar, March 27- April 1, 2011.

Cathedral of Applied Information SystemsUniversity of Information Technology and ManagementPoland

Janusz StarzykJanusz Starzyk

Motivated LearningMotivated Learning

Various pains, internal, and external signals compete for attention. Attention switching results from competition. Cognitive perception is aided by attention switching.

Definition: Motivated learning (ML) is pain based motivation, goal creation and learning in embodied agent. ML applies to EI working in a hostile environment. Machine creates abstract goals based on the pain

signals. It receives internal rewards for satisfying its goals

Reinforcement LearningReinforcement Learning Motivated Learning Motivated Learning External rewards Predictable Objectives set by designer Maximizes the reward

Potentially unstable

Learning effort increases with complexity

Always active

Internal rewards Unpredictable Sets its own objectives Solves minimax problem

Always stable

Learns better in complex environment than RL

Acts when needed

http://www.bradfordvts.co.uk/images/goal.jpg

Primitive Goal CreationPrimitive Goal Creation

- +

Pain

Dry soilPrimitive

level

opentank

sit on garbage

refillfaucet

w. can water

Dual pain

Reinforcing a proper action

Abstract Goal HierarchyAbstract Goal Hierarchy

Abstract goals are created to reduce abstract pains and to satisfy the primitive goals A hierarchy of abstract goals is created to satisfy the lower level goals

ActivationStimulationInhibitionReinforcementDifferenceNeedExpectation

- +

+

Dry soilPrimitive Level

Level I

Level IIfaucet

-

w. can

open

water

+

Sensory pathway(perception, sense)

Motor pathway(action, reaction)

Level IIItank

-

refill

Drought

Reservoir

Irrigate

Thirsty

Water

Drink Water

Primitive Needs Dirty

Wash in Water

Abstract Needs

Primitive needsPrimitive needs

0 50 100 150 200 250 3000

0.02

0.04

0.06

0.08

0.1

0.12Competing need signals

Iterative step

Nee

d si

gnal

leve

l

DirtyThirstyDroughtThreshold

Drought

Reservoir

Public Money

Irrigate

Spend Money to Build

Thirsty

Water

Drink Water

Spend Money to Buy

Primitive Needs

Well

Draw own Water

Dirty

Wash in Water

Abstract Needs

Abstract needsAbstract needs

Drought

Reservoir

Public Money

Tourists' Attractions

Irrigate

Spend Money to Build

Build Ecotourism

Thirsty

Water

Drink Water

Spend Money to Buy

Primitive Needs

Build Water Recreation

Wealthy Taxpayers

Rise Taxes

Well

Draw own Water

Dirty

Wash in Water

Well Building

Dig a Well

Abstract Needs

Ground Water

Water Supply

Abstract needsAbstract needs

Drought

Reservoir

Public Money

Tourists' Attractions

Irrigate

Spend Money to Build

Build Ecotourism

Thirsty

Water

Drink Water

Spend Money to Buy

Primitive Needs

Build Water Recreation

Policy

Develop Infrastructure

Wealthy Taxpayers

Rise Taxes

Well

Draw own Water

Dirty

Wash in Water

Well Building

Dig a Well

Abstract Needs

Employment Opportunities

Ground Water

Water Supply

Receive SalaryResource Management

and Planning

Management

Regulate Use

Planning

Abstract needsAbstract needs

6 levels of hierarchy Initially ML agent experiences similar

primitive pain signal Pp as RL agent. ML agent converges quickly to a stable

performance.

10 levels of hierarchy Initially RL agent experiences lower

primitive pain signal Pp than ML agent. RL agent’s pain increases when

environment is more hostile.

ML vs. RL ML vs. RL aagentgentss inin hierarchical hierarchical environmentenvironmentss

J.A. Starzyk, P. Raif, and A.-H. Tan, “Mental Development and Representation Building through Motivated Learning” , WCCI 2010 - Special Session on Mental Architecture and Representation, Barcelona, Spain, July 18-23, 2010.

Grid world problemGrid world problem

Four kinds of resources distributed over 25 x 25 grid.P. Raif, J.A. Starzyk, Motivated Learning In Autonomous Systems, submitted to IJCNN2011 - Special Session on Autonomous learning of object representation and control, San Jose, CA, July 31-Aug. 5, 2011.

Intelligence

Central executive

Attention and attention switching

Mental saccades

Cognitive perception

Cognitive action control

ConsciousnessConsciousness

Photo: http://eduspaces.net/csessums/weblog/11712.html

Computational Model of Conscious MachineComputational Model of Conscious Machine

Semantic memory

Sensory processors

Data encoders/ decoders

Sensory units

Motor skills

Motor processors

Data encoders/ decoders

Motor units

Emotions, rewards, and sub-cortical processing

Attention switching

Action monitoring

Motivation and goal processor

Planning and thinking

Episodic memory

Queuing and organization of episodes

Episodic Memory & Learning

Central Executive

Sensory-motor

Inspiration: human brainInspiration: human brainPhoto (brain): http://www.scholarpedia.org/article/Neuronal_correlates_of_consciousness

Attention switching

Action monitoring

Motivation and goal processor

Planning and thinking

Central Executive

Taskso cognitive perceptiono attentiono attention switchingo motivationo goal creation and selectiono thoughtso planningo learning, etc.

Central ExecutiveCentral Executive

Interacts with other units for o performing its tasks o gathering data o giving directions to other units

No clearly identified decision centerDecisions are influenced by

o competing signals representing motivations, pains, desires, plans, and interrupt signals

• need not be cognitive or consciously realizedo competition can be interrupted by attention switching signal

Attention switching

Action monitoring

Motivation and goal processor

Planning and thinking

Central Executive

Central ExecutiveCentral Executive

Attention Switching !Attention Switching !Dynamic process resulting from competition between

• representations related to motivations

• sensory inputs

• internal thoughts including

spurious signals (like noise).

blog.gigoo.org/.../

Input image

AB

C D

AB

C DA

B

C D

What Where

Visual SaccadesVisual Saccades

Mental SaccadesMental Saccades

This in turn activates memory traces in the global workspace area

that will be used for mental searches (mental saccades).

saccade

John

Input image

Episodic and associative memory network

his wife his house

his dogfriends

business

Spotlight on John

Frontal cortex

Mental saccade

wife house

dogfriends

business

Memory traces in frontal cortex

saccade

John

Input image

Episodic and associative memory network

his wife his house

his dogfriends

business

Spotlight on John

Frontal cortex

Mental saccade

wife house

dogfriends

business

Memory traces in frontal cortex

Selected part of the image resulting from an eye saccade.

Perceived input activates object recognition and associated areas of semantic and episodic memory.

Mental saccades in a conscious machineMental saccades in a conscious machine

Perceptual saccadesChanging perception

Changing environment

Associative memory

No

No

Action controlLoop 5

Loop 2

Perceptual saccadesChanging perception

Changing environment

Associative memory

No

No

Action control

Advancement of a goal?

Yes

Learning

Advancement of a goal?

Advancement of a goal?

Yes

Learning

Attention spotlight

Mental saccades

Continue search?

Yes

Loop 1

Attention spotlight

Mental saccades

Continue search?Continue search?

Yes

Loop 1

Plan action?

NoYes

Action?

Yes

No

Changing motivation

Loop 3

Loop 4

Plan action?Plan action?

NoYes

Action?Action?

Yes

No

Changing motivation

Loop 3

Loop 4

Loop 5

Loop 2

Action and subgoal planningAction and subgoal planning

Intended action

Induced pain

Dual pain

Perception

Pain reduction Next mental saccade

Perform action

Learning

Pain

Environment

Decide action

Attention spotlight

Desired item

Memory

Action controlAction control

Predicted changes known

Pain increase

Predicted changes

Intended action

Associative memory

Cognitive action control

Lower level action control

Lower level action control

Lower level action control

Action?

Cognitive abort

A Mechanism for Learning, Attention A Mechanism for Learning, Attention Switching, and Cognition: SummarySwitching, and Cognition: Summary

A mechanism of switching attention is fundamental for building cognitive machines.Attention switching is a dynamic process resulting from competition between goals, representations, sensory inputs, and internal thoughts.Motivated learning provides a mechanism for creation of abstract goals and continuous goal oriented motivationMental saccades of the working memory are fundamental for cognitive thinking, attention switching, planning, and action monitoring

http://www.inspirationfalls.com/the-key-to-success-quotes/key-to-all-success-concepts-1/

Motivations for actions are physically distributedo competing pain (need) signals are generated in various parts

of machine’s mind Before a winner is selected, machine does not interpret the meaning of the competing signals Cognitive processing is predominantly sequential

o winner of the internal competition is an instantaneous director of the cognitive thought process

Top down supervision of perception, planning, internal thought or motor functions

o results in conscious experience• decision of what is observed and where is it • planning how to respond

o a train of such experiences constitutes consciousness

A Mechanism for Learning, Attention A Mechanism for Learning, Attention Switching, and Cognition: SummarySwitching, and Cognition: Summary

ConclusionsConclusions1.Consciousness is computational2.Motivated intelligent machines can be conscious

Questions ??Questions ??

Photo: http://bajan.wordpress.com/2010/03/03/dont-blame-life-blame-the-way-how-you-live-it/

ReferencesReferences P.A.O. Haikonen, “The cognitive approach to conscious machines”. UK: Imprint Academic, 2003. J. Bach, “Principles of Synthetic Intelligence PSI: An Architecture of Motivated

Cognition”, Oxford Univ. Press, 2009. B. J. Baars “A cognitive theory of consciousness,” Cambridge Univ. Press, 1998. A. Sloman, "Developing concept of consciousness," Behavioral and Brain

Sciences, vol. 14 (4), pp. 694-695, Dec 1991. J. Schmidhuber, “Curious model-building control systems,” Proceedings Int. Joint

Conf. Neural Networks, Singapore, vol. 2, pp. 1458–1463, 1991. B. Bakker and J. Schmidhuber, “Hierarchical Reinforcement Learning with

Subpolicies Specializing for Learned Subgoals,” in Proc. of the 2nd Int. Conf. on Neural Networks & Computational Intelligence, Switzerland, pp.125-130, 2004.

A. Barto, S. Singh, and N. Chentanez, “Intrinsically motivated learning of hierarchical collections of skills, Proc. 3rd Int. Conf. Development Learn., San Diego, CA, pp. 112–119, 2004.

J. A. Starzyk, "Motivation in Embodied Intelligence"  in Frontiers in Robotics, Automation and Control, Oct. 2008, pp. 83-110.

J.A. Starzyk, “Motivated Learning for Computational Intelligence,” in Computational Modeling and Simulation of Intellect. ed. B. Igelnik, IGI Publ, 2011.

Photo: http://s121.photobucket.com/albums/o209/TiTekty/?action=view&current=hist_sci_image1.jpg

Embodied IntelligenceEmbodied Intelligence

– Mechanism: biological, mechanical or virtual agent

with embodied sensors and actuators– EI acts on environment and perceives its actions– Environment hostility is persistent and stimulates EI to act– Hostility: direct aggression, pain, scarce resources, etc– EI learns so it must have associative self-organizing memory– Knowledge is acquired by EI

Definition Embodied Intelligence (EI) is a

mechanism that learns how to minimize hostility of its environment

Embodiment

Actuators

Sensors

Intelligence core

channel

channel

Embodiment

Sensors

Intelligence core

Environment

channel

channelActuators

Embodiment

Actuators

Sensors

Intelligence core

channel

channel

Embodiment

Sensors

Intelligence core

Environment

channel

channelActuators

Embodiment of a MindEmbodiment of a Mind

Embodiment is a part of the environment that EI controls to interact with the rest of the environment

It contains intelligence core and sensory motor interfaces under its control

Necessary for development of intelligence

Not necessarily constant or in the form of a physical body

Boundary transforms modifying brain’s self-determination