Timo Honkela: Linking Cognitive Systems, Digital Humanities and Brain Cancer Experiences

Timo Honkela, Neuroscience Seminar presentation, 18.9.2015

Timo Honkela

Neuroscience seminar series, Host Prof. Eero CastrénUniversity of Helsinki, 18 Sep 2015

Linking Cognitive Systems, Digital Humanities and Brain

Cancer Experiences

[email protected]


Speaker'sBack ground


Summary of study & work career

● M.Sc. on human-oriented information systems development at University of Oulu

● Sitra's Kielikone project● VTT Information Technology● Neural Networks Research Center, Helsinki University of

Technology, PhD ● Media Lab, University of Art and Design Helsinki,

professor● TKK > Aalto University, head of Cognitive Systems group● University of Helsinki and National Library of Finland,

professor


Hand-crafted,symbol manipulation

based AI


Early personal experiences onrule-based natural language processing

● H. Jäppinen, T. Honkela, H. Hyötyniemi & A. Lehtola (1988):A Multilevel Natural Language Processing Model. Nordic Journal of Linguistics 11:69-87.

What is the turnover of the ten largest stock exchange companies in forestry?

Morphological analysis

Dependency parsing

Logical analysis

Database query formation

Result from the SQL database


Neural network /machine learning based AI


Classical example: Learning meaning from context:

Maps of words in Grimm fairy tales

Honkela, Pulkki & Kohonen 1995


Chemistry

Natural sciencesand engineering

Bio- andenvironmentalsciences

Health

Culture andsociety

Map of Finnish Science


Multimodallygrounded AI


Labeling movements: Associatinghigh-dim. kinesthetic time series

with linguistic labels

Förger & Honkela 2014


RUNNING

WALKING

LIMPING

JOGGING

Förger & Honkela 2014


CognitiveSystems


Ambiguity (homography & polysemy) and contextuality: case “ALUSTA”

● “ALUSTA”

"alku" N ELA SG

"alusta" N NOM SG

"alustaa" V PRES ACT NEG

"alustaa" V IMPV ACT SG2

"alustaa" V IMPV ACT NEG SG

"alunen" N PTV SG

"alus" N PTV SG

FINTWOL: Finnish Morphological Analyser Copyright © Kimmo Koskenniemi & Lingsoft Oy 1995 – 2012http://www2.lingsoft.fi/cgi-bin/fintwol

Alusta

Monta alusta

Näin monta alusta

Näin monta alustasatamassa


Ambiguity (homography & polysemy) and contextuality: case “ALUSTA”

● “ALUSTA”

"alku" N ELA SG

"alusta" N NOM SG

"alustaa" V PRES ACT NEG

"alustaa" V IMPV ACT SG2

"alustaa" V IMPV ACT NEG SG

"alunen" N PTV SG

"alus" N PTV SG

FINTWOL: Finnish Morphological Analyser Copyright © Kimmo Koskenniemi & Lingsoft Oy 1995 – 2012http://www2.lingsoft.fi/cgi-bin/fintwol

Alusta

Monta alusta

Näin monta alusta

Näin monta alustasatamassa alastaivaalta

http

://fa

vim

.com

/imag

e/92

863/


Ambiguity (homography & polysemy) and contextuality: case “GET”

● “ S: (v) get, acquire (come into the possession of something concrete or abstract) "She got a lot of paintings from her uncle"; "They acquired a new pet"; "Get your results the next day"; "Get permission to take a few days off from work"● S: (v) become, go, get (enter or assume a certain state or condition) "He became annoyed when he heard the bad news"; "It must be getting more serious"; "her face went red with anger"; "She went into ecstasy"; "Get going!"● S: (v) get, let, have (cause to move; cause to be in a certain position or condition) "He got his squad on the ball"; "This let me in for a big surprise"; "He got a girl into trouble"● S: (v) receive, get, find, obtain, incur (receive a specified treatment (abstract)) "These aspects of civilization do not find expression or receive an interpretation"; "His movie received a good review"; "I got nothing but trouble for my good intentions"● S: (v) arrive, get, come (reach a destination; arrive by movement or progress) "She arrived home at 7 o'clock"; "She didn't get to Chicago until after midnight"● S: (v) bring, get, convey, fetch (go or come after and bring or take back) "Get me those books over there, please"; "Could you bring the wine?"; "The dog fetched the hat"● S: (v) experience, receive, have, get (go through (mental or physical states or experiences)) "get an idea"; "experience vertigo"; "get nauseous"; "receive injuries"; "have a feeling"● S: (v) pay back, pay off, get, fix (take vengeance on or get even) "We'll get them!"; "That'll fix him good!"; "This time I got him"● S: (v) have, get, make (achieve a point or goal) "Nicklaus had a 70"; "The Brazilian team got 4 goals"; "She made 29 points that day"● S: (v) induce, stimulate, cause, have, get, make (cause to do; cause to act in a specified manner) "The ads induced me to buy a VCR"; "My children finally got me to buy a computer"; "My wife made me buy a new sofa"● S: (v) get, catch, capture (succeed in catching or seizing, especially after a chase) "We finally got the suspect"; "Did you catch the thief?"● S: (v) grow, develop, produce, get, acquire (come to have or undergo a change of (physical features and attributes)) "He grew a beard"; "The patient developed abdominal pains"; "I got funny spots all over my body"; "Well-developed breasts"● S: (v) contract, take, get (be stricken by an illness, fall victim to an illness) "He got AIDS"; "She came down with pneumonia"; "She took a chill"● S: (v) get (communicate with a place or person; establish communication with, as if by telephone) "Bill called this number and he got Mary"; "The operator couldn't get Kobe because of the earthquake"● S: (v) make, get (give certain properties to something) "get someone mad"; "She made us look silly"; "He made a fool of himself at the meeting"; "Don't make this into a big deal"; "This invention will make you a millionaire"; "Make yourself clear"● S: (v) drive, get, aim (move into a desired direction of discourse) "What are you driving at?"● S: (v) catch, get (grasp with the mind or develop an understanding of) "did you catch that allusion?"; "We caught something of his theory in the lecture"; "don't catch your meaning"; "did you get it?"; "She didn't get the joke"; "I just don't get him"● S: (v) catch, arrest, get (attract and fix) "His look caught her"; "She caught his eye"; "Catch the attention of the waiter"● S: (v) get, catch (reach with a blow or hit in a particular spot) "the rock caught her in the back of the head"; "The blow got him in the back"; "The punch caught him in the stomach"● S: (v) get (reach by calculation) "What do you get when you add up these numbers?"● S: (v) get (acquire as a result of some effort or action) "You cannot get water out of a stone"; "Where did she get these news?"● S: (v) get (purchase) "What did you get at the toy store?"● S: (v) catch, get (perceive by hearing) "I didn't catch your name"; "She didn't get his name when they met the first time"● S: (v) catch, get (suffer from the receipt of) "She will catch hell for this behavior!"● S: (v) get, receive (receive as a retribution or punishment) "He got 5 years in prison"● S: (v) scram, buzz off, fuck off, get, bugger off (leave immediately; used usually in the imperative form) "Scram!"● S: (v) get (reach and board) "She got the bus just as it was leaving"● S: (v) get, get under one's skin (irritate) "Her childish behavior really get to me"; "His lying really gets me"● S: (v) get (evoke an emotional response) "Brahms's `Requiem' gets me every time"● S: (v) catch, get (apprehend and reproduce accurately) "She really caught the spirit of the place in her drawings"; "She got the mood just right in her photographs"● S: (v) draw, get (earn or achieve a base by being walked by the pitcher) "He drew a base on balls"● S: (v) get (overcome or destroy) "The ice storm got my hibiscus"; "the cat got the goldfish"● S: (v) perplex, vex, stick, get, puzzle, mystify, baffle, beat, pose, bewilder, flummox, stupefy, nonplus, gravel, amaze, dumbfound (be a mystery or bewildering to) "This beats me!"; "Got me--I don't know the answer!"; "a vexing problem"; "

This question really stuck me"● S: (v) get down, begin, get, start out, start, set about, set out, commence (take the first step or steps in carrying out an action) "We began working at dawn"; "Who will start?"; "Get working as soon as the sun rises!"; "The first tourists began to arrive

in Cambodia"; "He began early in the day"; "Let's get down to work now"● S: (v) suffer, sustain, have, get (undergo (as of injuries and illnesses)) "She suffered a fracture in the accident"; "He had an insulin shock after eating three candy bars"; "She got a bruise on her leg"; "He got his arm broken in the scuffle"● S: (v) beget, get, engender, father, mother, sire, generate, bring forth (make (offspring) by reproduction) "Abraham begot Isaac"; "John fathered four daughters"

Wor

dNet

3.1


Effect of context: Color naming

Human vision: rods, cones,...Physical reasons for colorContextuality of naming

red winered skinred shirt

Hardin

Gärdenfors


User-specific difficulty measure All users have their own background knowledge and

vocabulary: different texts are difficult for different people -> need for user modeling

Paukkeri, Ollikainen & Honkela, 2013


User-specific difficulty measure



User-specific difficulty measure



Tensor-based analysis ofsubjectivity


GICA: Grounded IntersubjectiveConcept Analysis

Sanat,fraasit,tulkinnat tms.

Kontekstit

Yksilöt


The word “health” inState of the Union Addresses

Subjects on objects in contexts: Using GICA method to quantify epistemological subjectivity. Timo Honkela, Juha Raitio, Krista Lagus, Ilari T. Nieminen, Nina Honkela, and Mika Pantzar.Proc. of IJCNN 2012.


Implications of multiperspective knowledge engineering

● Descriptions of contents need to bestandardized to a lesser degreebecause systems learn to createmappings between different conceptual systems

● In the future, machines can facilitate meaning negotiations between, e.g., experts of different disciplines or between experts and laypersons


DigitalHumanities

Case: Historical Newspaper Collectionof the National Library of Finland


Digital humanities

● Research within humanities with the help of computers– Digital resources

– Computational models

● Basic motivation– One can already fly to moon and

build sophisticated factory products

– The most important open questionsin the world are related to humanitiesand social sciences


Digital Computational

Humanities

Contentstorage and

transfer

Contentanalysis


Accessing and analyzing digital resources


Resources

Content andinformationprofessional

Users ofthe contents

(professionalsand lay people)

Machine learningand

pattern recognitionsystems

Formal metadata

Languagetechnology

resources andsystems

Other forms of description


Resources

Users ofthe contents

(professionalsand lay people)

Other forms of description

Crowdsourcing

Importanceof openness


Historical newspaper collection

● The National Library of Finland has digitized a large proportion of the historical newspapers published in Finland between 1771 and 1910 (Bremer-Laamanen 2001, 2005).

● This collection contains approximately 1.95 million pages in Finnish and Swedish

● According to Legal Deposit law, the National Library of Finland receives a copy of each newspaper and magazine published in Finland.


Search interface

http://digi.kansalliskirjasto.fi


FIN-CLARIN corpus

www.kielipankki.fi


Challenges of pattern recognition


OCR Challenges

● Regardless of recent development of the OCR software, there are still challenges with it, as some material is very old, with – varying paper and print quality,

– varying number of columns and layout patterns,

– different languages (mainly Finnish and Swedish but also French, German, etc.), and

– varying font types (fraktur and antiqua)


zzhdysvautki Yhdyspankki

v, u, p ? u, n, ll ?




A very long tail of low frequency forms...


taioafliftiutpn tawallisuuden


Similarity diagram of Fraktur letter shapes(a self-organizing map)


Areas of further analysis

● Multidimensional sentiment analysis● Analysis of social and

historical context● Intercultural and

multilingual analysis● Analysis of points of view● Comparison with other data sets such as modern

newspapers or social media discussions ● Analysis of subjective

understanding


BrainCancer


Einsiedeln,Switzerland,May 2014

The right side of my visual field disappeared

Helsinki,May 2014

TIA (transientischemic attack)

negative

Somethins white behind,maybe an old stroke.


Mikkeli,September 2014

I am completely exhaustedand my right hand

does not movenormally

Helsinki,September 2014

A sick leave is neededfor your burnout

after working 70 hoursper week


Espoo,November 2014

My husband, normallyan intelligent professor is “our of this world”. I takehim to the hospital now!

Espoo, Jorvi,Nov 2014

Normally we would forceyou to go through

your occupational doctor butas you don't know which year it is...

You have a largetumor in your brain


Treatments

● Surgical opetation● Radiation therapy● Cytostatic drugs

● Good nursing● Psychological consultations● Physiotherapeutic consultations

Supportive action

Other medical actions

● Neuro-oftalmologic consult.● Neurologic consultations● Psychiatric consultations

● MRI


Treatments

● Surgical operation● Radiation therapy● Cytostatic drugs

● Good nursing● Psychological consultations● Physiotherapeutic consultations

Supportive action

Other medical actions

● Neuro-oftalmologic consult.● Neurologic consultations● Psychiatric consultations

● MRI


Operation 1st of Dec 2014: A completesuccess but unfortunately malignant ...

Operated by world famous prof. Hernesniemi


Expressingmy deepestgratitude.

Wishing morefunding whereverneeded and better health careinformation systems.


Radiationtherapy 6 weeks plus low dose ofcytostatic drugs

Six months of cytostatictreatment 5 days/month

300 mg/day


Plan for the radiation therapycarefully avoidingthe brain stem.


MRI- Dec 2014 (om the left)- Spring 2015

- Left visual cortex defected- Additional problems caused by inflammation, cortisone helped quickly


Blood tests to check if treatment can be continued


Visual fieldimpairment


After the operation:“Random” top-down simulation

Nowadays:“Data-driven” simulation


Own actions

● Optimistic attitude● Enjoying life

excl. dark moments● Family support● Social relations● Peer support● Physical exercise● Mental exercises● Therapeutic effect of

music

● “Ten changes”regime

● Visualizationexercising experiment

● Nutrition:healthy & enjoyable

● Light work (oppositeto the old scheme)

● Openness


Colleagues visiting Jorvi two days before operation. Discussions strictly on Digital Humanities.


Social mediaforum

Etelä-Suomensyöpäyhdistys


Dec 14, 2014

Dec

26,

201

4

Jun, 2015

Sep 9, 20 15

Aug, 2015



Christmasporridge


Work experiment

● 20 hours per week● Supported by KEVA● Started 1st of September● Basically 4 hours per day,

partly at home


Helsinki Challengepitch Sep 3, 2015


Planning next steps


Ideas,opportunies


Open data volunteers

● People could donate their health and life style data for research

● This attitude could be promoted as a primary moral choice that helps other people

● If widely adopted, this kind of practice requires strongly integrity from the people who use the data


High-dimensional health research

● It would be useful to study very high-dimensional data sets with an open minded attitude– Gene data

– Life style factors (exercise, nutrition, etc.)

– Emotions

– Environment (e.g. chemical risk factors)

● These should be analyzed with latest data-driven statistical machine learning methods


“Mind-body problem”

● Philosophers have considered different options on how mind and body are related or if they are the same

● Medical doctors often seem to set this issue aside, refer to the plasebo effect, or send the patient to a psychiatrist

● However, there seems to be more in this issue that would require more careful research than what has been conducted so far


Adaptive neuro-physiological systems

● An opportunity woud be to conduct interdisciplinary research in which human health would be studies concurrently from multiple perspectives

● The data would be gathered at multiple levels of detail and abstraction including gene data, immunological data, hormonal process, emotional factors, etc.

● This data could be collected including bothobjective measurements and subjectiveassessments


Measuring the subjective

● It is well known that the variation in how people describe their condition is high even if they seem to suffer of the same condition

● Medical doctors may have developed some kind of sensitivity to hypocondria but in general this modeling is most likely quite superficial

● There are new methods that could be used tomodel the subjectivity in a more fine grained manner, helping both the medical professionals and the patients


Thank you foryour attention!

Health & Medicine

Timo Honkela: Linking Cognitive Systems, Digital Humanities and Brain Cancer Experiences