Future of Artificial Intelligence- Memory, Knowledge, and Language

Hang Li

Noah’s Ark Lab

Huawei Technologies

Global Artificial Intelligence Technology ConferenceBeijing, May 21, 2017

Outline

• Memory and Intelligence

• Intelligent Information and Knowledge Management System

• Neural Symbolic Processing

• Related Work

• Our Work

• Summary

Life without Memory - Tragic Story of Clive Wearing

• A musician in UK

• Suffering from amnesia (失忆症), after

contracting encephalitis

• Symptom: lacks ability to form new memories, and also cannot recall most of his past memories

• Can speak, eat, dress himself, walk, ride bike, even play the piano

• Can remember his wife’s name, but cannot remember his daughter’s name

• Only has moment-to-moment consciousness

Clive Wearing

Video

Human Brain

Model of Human Brain

Modified from Frank Longo 2010

Sensory Register• vision 1 sec• auditory 5 sec

Short-term Memory• 18-30 sec• 7±2

• displacement

Long-term Memory

• synapse mod• limitless cap• declarative and non-declarative

Attention Consolidation

RetrievalCentral Executive Unit• planning• conscious thought

visualauditorykinaestheticolfactorygustatory

Characteristics of Long-term Memory

• Storing information and knowledge (not data)

• New content can be continuously added

• New content is associated with existing content

Current AI Systems Are All ‘Memoryless’

• Do not have long-term memory

• Repeatedly utilize learned functions

• Cf., Clive Wearing

)(

)(

xg

xf

AlphaGo

Self Driving Car)(xf

)(xg

)(xh

Loop of functions Finite state of functions

AI Systems Will Evolve to Next Level, When Having Long Term Memory

• Do AI Systems Have Consciousness?

• There is no consensus on consciousness

• Kaku: Consciousness means that the system changes internal states in interaction with environment

• In that definition, thermostat, sunflowers, etc have ‘consciousness’; AI systems also have ‘consciousness’

• AI systems will have `continuous consciousness’ with long term memory

Michio Kaku

Outline

• Memory and Intelligence

• Intelligent Information and Knowledge Management System

• Neural Symbolic Processing

• Related Work

• Our Work

• Summary

Information and Knowledge

Wisdom

Knowledge

Information

Data

• Information: facts about people, objects, and events

• Knowledge: theoretical or practical understanding of a subject

• Plato: Knowledge = Justified True Belief (JTB)

• No clear boundary between information and knowledge

• Most of our information and knowledge is acquired and learned from others

Intelligent Information and Knowledge Management System with Long Term Memory

Language Processing Unit

Short-term Memory

Long-term Memory

Consolidation

Central Executive Unit

Analysis

Q2

A1

Learning Phase

Q1

A2

… …

Intelligent Information and Knowledge Management System with Long Term Memory

Language Processing Unit

Short-term Memory

Long-term Memory

Retrieval

Central Executive Unit

Analysis

Q

A

Use Phase

Characteristics and Current Status

• Continuously accumulates information and knowledge

• Properly performs question answering in natural language,

– Answers when it knows

– Says “I don’t know”, when it does not know

• Ideally, system is fully automatically constructed without human involvement

• Becomes intelligent assistant of human

– Note that computer has two powerful capabilities, computing and storage

• Currently only partially realized, cf., search engine

Outline

• Memory and Intelligence

• Intelligent Information and Knowledge Management System

• Neural Symbolic Processing

• Related Work

• Our Work

• Summary

Challenges

• Knowledge is not categorical

• Language is polysemous and synonymous

More light!

Knowledge Is Not Categorical- Example: Bachelor

• Bachelor: unmarried adult male

• How to judge the following?

– Unmarried father of child

– Man having fake marriage

– 17 year old high school student

– 17 year old playboy

– Homosexual lovers

– Arabic man with two wives to meet another fiancee

– Bishop

• From Terry Winograd

• Arthur has been living happily with Alice for the last five years. They have a two-year-old daughter and have never officially married.

• Bruce was going to be drafted, so he arranged with his friend Barbara to have a justice of the peace marry them so he would be exempt. They have never lived together. He dates a number of women, and plans to have the marriage annulled as soon as he finds someone he wants to marry.

• Charlie is 17 years old. He lives at home with his parents and is in high school.

• David is 17 years old. He left home at 13, started a small business, and is now a successful young entrepreneur leading a playboy's lifestyle in his penthouse apartment.

• Eli and Edgar are homosexual lovers who have been living together for many years.

• Faisal is allowed by the law of his native Abu Dhabi to have three wives. He currently has two and is interested in meeting another potential fiancee.

• Father Gregory is the bishop of the Catholic cathedral at Groton upon Thames.

Language Is PolysemousExample: Climb

• The boy climbed the tree.• The locomotive climbed the

mountainside.• The plane climbed to 30,000 feet.• * Smoke climbed from a chimney.• * An elevator climbed from one

floor to another.• The temperature climbed into the

90s.• Prices are climbing day by day.• The boy climbed down the tree

and over the wall.• We climbed along the cliff edge.• * The locomotive climbed over the

mountain.• He climbed out of a sleeping-bag.

18

• Climb: motion from lower level to higher level, along a path, by laborious manipulation of limbs

• Features: [ascend] [clamber]

• Climb is polysemous category consisting of several senses

• The senses are related through meaning chain A-B-C-D

• From Charles Fillmore

Language Is SynonymousExample: Distance between Sun and Earth

• distance from earth to the sun

• distance from sun to earth

• distance from sun to the earth

• distance from the earth to the sun

• distance from the sun to earth

• distance from the sun to the earth

• distance of earth from sun

• distance between earth sun

• "how far" earth sun

• "how far" sun

• "how far" sun earth

• average distance earth sun

• average distance from earth to sun

• average distance from the earth to the sun

• distance between earth & sun

• distance between earth and sun

• distance between earth and the sun

• how far away is the sun

from earth

• how far away is the sun

from the earth

• how far earth from sun

• how far earth is from

the sun

• how far from earth is

the sun

• how far from earth to

sun

• how far from the earth

to the sun

• distance between sun

and earth19

Combination of Neural Processing and Symbolic Processing

Easy to Interpret

Easy to Manipulate

Able to Deal with

Uncertainty

Robust to Noise

Symbolic Representation Neural Representation

Neural Symbolic Processing

Neural Symbolic Processing for Information and Knowledge Management

Central Executive Unit

Q

A

Short-Term Memory Long-Term Memory

Q A

Knowledgein symbolic representation & neural representation

Sym.Neu.

Language ProcessingModel

Encoder

Decoder

Outline

• Memory and Intelligence

• Intelligent Information and Knowledge Management System

• Neural Symbolic Processing

• Related Work

• Our Work

• Summary

Never Ending Language Learning (NELL)

• Task

– Initial ontology, few examples of each category, the web, occasional interaction from humans

– Extract more facts from web

– Learn to read better than before

• System

– Knowledge base with 15 million candidate beliefs

• Technologies

– Coupled semi-supervised learning, automatic discovery of new coupling constraints, automatic extending of ontology, staged curriculum

Mitchell et al. 2015

Memory Networks

• Long term memory + inference

• Model is learned

• Can answer factoid questions

• Acc = 40%+

• Example– John is in the playground.

– Bob is in the office.

– John picked up the football.

– Bob went to the kitchen.

– Q: where is the football?

– A: playground

memory

pre-process

generate

matchq

a

i

o

im

Weston et al. 2014

Differentiable Neural Computers

• DNC = neural network + external memory (matrix)

• Memory represents complex data structures

• Neural network, learned from data and supervised learning, controls access to memory

• Memory heads use three forms of differentiable attention

• Resembling mammalian hippocampus functions

Graves et al. 2016

Outline

• Memory and Intelligence

• Intelligent Information and Knowledge Management System

• Neural Symbolic Processing

• Related Work

• Our Work

• Summary

Researchers

Zhengdong Lu Lifeng ShangXin Jiang

Question Answering from Knowledge Graph

(Yao-Ming, spouse, Ye-Li)

(Yao-Ming, born, Shanghai)

(Yao-Ming, height, 2.29m)

… …

(Ludwig van Beethoven, place of

birth, Germany)

… …

Knowledge Graph

Q: How tall is Yao Ming?

A: He is 2.29m tall and is visible from space.(Yao Ming, height, 2.29m)

Q: Which country was Beethoven from?

A: He was born in what is now Germany.(Ludwig van Beethoven, place of birth, Germany)

Question Answering

System

Q: How tall is Liu Xiang? A: He is 1.89m tall

Learning System

Answer is generated

GenQA

Encoder

Decoder

Short-Term Memory Long-Term MemoryLanguage Processing Module

Q

A

Q’

A’

Triples in symbolic representations (indexed) & neural representations

IndexTriples

Encoder creates question representation, decoder generates answer

Matches and retrieves most relevant answer representation

End-to-End Training

Question Answering from Relational Database

Relational Database

Q: How many people participated in the game in Beijing?

A: 4,200

SQL: select #_participants, where

city=beijing

Q: When was the latest game hosted?

A: 2012

SQL: argmax(city, year)

Question Answering

System

Q: Which city hosted the longest Olympic game before the game in Beijing? A: Athens

Learning System

year city #_days #_medals

2000 Sydney 20 2,000

2004 Athens 35 1,500

2008 Beijing 30 2,500

2012 London 40 2,300

Neural Enquirer End-to-End Training

Encoder

Short-Term Memory

Long-Term Memory

Language Processing Module

Q

A

Q’

A’

Features and values are in symbolic representations and neural representations

Encoder creates question representation, decoder simply returns answer

Matches question representation to table representations to find answer

Decoder

Matching

Neural Enquirer

• Five layers, except last layer, each layer has reader, annotator, memory • Reader fetches important representation for each row• Annotator encodes result representation for each row

Video

Outline

• Memory and Intelligence

• Intelligent Information and Knowledge Management System

• Neural Symbolic Processing

• Related Work

• Our Work

• Summary

Summary

• Long term memory is indispensable for human intelligence

• AI systems will evolve to next level with long term memory

• Intelligent Information and Management System

– Can automatically acquire information and knowledge

– Can correctly answer questions from humans

• This should be most important topic for research in AI

• Neural Symbolic Processing should be most promising approach

• Recent research is making progress

• Many open questions and challenges

Thank You!