International Journal of Man-Machine Studies Volume 27 Issue 4 1987 [Doi 10.1016%2Fs0020-7373%2887%2980004-4] Colleen Crangle; Patrick Suppes -- Context-fixing Semantics for Instructable

8/10/2019 International Journal of Man-Machine Studies Volume 27 Issue 4 1987 [Doi 10.1016%2Fs0020-7373%2887%29800

http:///reader/full/international-journal-of-man-machine-studies-volume-27-issue-4-1987-doi-1010162fs0020-7373 1/30

I n t J. Man-Machine Studies(1987) 2 / 371-400

Co ntex t f ix ing sem ant ics fo r ins t ruc tab le robo t s

C O L L E E N C R N G L E N D P T R I C K S U P P E S

Institute fo r Ma them atical Studies in the Soc ial Sciences, Ventura H all, Sta nfo rdUniversity, Stanford , C A 94 305, U .S.A .

Received 23 July 1986 and in revised fo rm 22 Ap r i l 1987)

Ins t ructable robo ts must be able to in terpre t a wide range of ordinary natura l -l anguage commands . Th i s paper p resen t s an approach to the in te rp re ta t ion o fcommands tha t takes in to account the context in which the commands are g iven. I tshows how the prec ise in terpre ta t ion of many ordinary Engl ish words can be f ixedonly wi th in the i r context of use and not before . I t examines the ro le of theperceptual s i tua t ion in f ix ing tha t in terpre ta t ion , the ro le of the cogni t ive andperceptual funct ioning of the robot , and the ro le of the immedia te l inguis t icsu r round . Ou r a pproach d raws o n the mo de l - theore t i c t r ad it ion in semant i cs in tha ti t def ines a se t of models in te rms of which the Engl ish commands to the robot arein terpre ted . At the same t ime, i t uses a procedura l semant ics for the lexicon, thusaddress ing the ques t ion of how the robot can use the ins t ruct ion i t i s g iven toperform the task descr ibed by tha t ins t ruct ion . Examples are drawn pr imar i ly f romins t ruc t ion in e l em enta ry ma themat i c s , O the r examp les come f rom our r ecen t workwith a robo t ic a id fo r the p hysica lly d isab led . t

1 I n t r o d u c t i o n

T h e c o n t e x t i n w h i c h a c o m m a n d is g i v e n s tr o n g l y d e t e r m i n e s t h e m e a n i n g o f t h a tc o m m a n d . T h i s f a c t h as l o n g b e e n r e c o g n i z e d f o r ce r t a in c l as se s o f w o r d s . I n d e x ic a lp r o n o u n s s u c h a s I a n dyou , a n d a d v e r b s o f p l a c e a n d t i m e s u c h a shere, now, a n dtoday, a r e c l e a r ly fi x e d b y t h e c o n t e x t o f t h e i r u s e , a s a r e a n a p h o r i c p r o n o u n s s u c has it a n d she. W h a t h a s n o t b e e n o b v i o u s u p t o n o w is t h a t m a n y o r d i n a r y w o r d sh a v e t h e i r p r e c i s e m e a n i n g f ix e d b y t h e i r c o n t e x t o f u s e . T h i s p a p e r p r e s e n t s a n

a p p r o a c h t o t h e i n t e r p r e t a t i o n o f n a t u r a l - la n g u a g e c o m m a n d s t h a t e x p lo i t s t h ee s s en t ia l ly c o n t e x t u a l n a t u r e o f o r d i n a r y E n g l i sh w o r d s .I t p a y s s p e c i a l a t t e n t i o n t o t h e p e r c e p t u a l s i t u a t i o n i n w h i c h t h e w o r d s a r e u s e d , i n

c o n t r a s t t o a n a l y s e s s u c h a s t h o s e o f G r o s z a n d S i d n e r ( 1 9 85 ) t h a t e x p l o r e o t h e ra s p e ct s o f t h e d i s c o u r s e c o n t e x t , s u c h a s s p e a k e r ' s i n t e n t io n s . A n e m p h a s i s o n t h ep e r c e p t u a l s it u a t i o n in la n g u a g e u n d e r s t a n d i n g a n d u s e is n o t n e w. M o r e t h a n t e ny e a r s a g o M i l l e r a n d J o h n s o n - L a i r d ( 1 9 76 ) , w r i t in g a b o u t l a n g u a g e a n d p e r c e p t i o n ,o f f e r e d p e r c e p t u a l p r o c e d u r e s f o r m a n y d i f f e r e n t c la s se s o f w o r d s . H o w e v e r , t h e i rw o r k d i d n o t s h a r e o u r s t r o n g s e m a n t i c c l a i m t h a t w o r d s i n g e n e r a l d o n o t h a v ed e t e r m i n a t e m e a n i n g b u t t h a t t h e i r s e m a n t i c s i g ni fi ca n c e is f ix e d b y th e a c t u a lo c c a s i o n s o f t h e i r u s e a n d t h a t t h e p e r c e p t u a l s i t u a t i o n h e l p s s p e c i fy t h o s e o c c a s io n so f u s e .

A p p e a l s t o c o n t e x t a r e a ls o n o t n e w a n d h a v e r e c e n t l y f o u n d c o n v i n c i n ge x p r e s s i o n i n Wi n o g r a d a n d F l o r e s ( 1 9 8 6) . W h a t is s ig n i fi ca n t a b o u t t h e w o r k

t The work on the robotic aid is funded by the Rehabilitation Research and Development Center ofthe Veterans Adm inistration, VA M edical Center, Pa lo Alto, California.

3710020-7373/87/100371 + 28503.00/0 9 1987 Ac adem ic Press Lim ited



37 C . C R A N G L E A N D P. S U P P E S

repor ted here i s the par t icu la r subs tance g iven to these appea ls . Al l na tura l -language c om m and s a re in te rpre te d re la t ive to a g iven c lass of perc eptu a l s i tua tionsand a g iven c lass of l anguage users . In m ode l - theore t ic t e rm s , w e def ine a se t M of

mode l s t ha t cha rac t e r i ze t he pe rcep tua l s i t ua t i ons i n wh ich the commands a r e g ivenand the p e rcep tua l and cogn i t ive funct ion ing o f t he agen t t o whom the comm andsare ad dresse d . O ur fund am enta l p rem ise i s tha t b y f ix ing the c lass of m ode ls wi th inwh ich the language m akes s en se , t he spec if ic and a pp rop r i a t e mean ing o f manyord inary Engl i sh wo rds is f ixed , and n ot o therwise . This pa per i s o rganize d a roundthe p re sen t a t i on o f tha t s e t M o f mode l s and an ex t en ded d i scus sion o f howcommands a r e i n t e rp re t ed r e l a t i ve t o i t .

The work r epo r t ed he re f al ls w i thin a l arge r p rog ram o f r e sea rch on in s truc tab l erobots . I t d raw s d i rec t ly on the p r ior w ork of Ma as and Su ppe s (1983 , 1984 , 1985) indeve loping a na tura l - langu age in te r face to an ins t ruc tab le robot . T he language usedi s t he o rd ina ry l anguage one wou ld u se t o t each someone , ch i ld o r robo t , s t anda rdar i thmet ic tasks such as co lumn addi t ion . Whi le th i s paper fea tures the a r i thmet icrobo t i n t roduc ed by Maas and Suppes , o u r cu r r en t w ork on in s truc t ab le rob o t s isbased on the omnid i r ec t i ona l robo t unde r deve lopmen t a s an a id fo r t he phys i ca l l yd isab led (M icha lowsk i Le i fe r, 1983; M icha lowski , 1986; Crang le , Su ppe sM icha lowski , 1987; M icha lowsk i , Cran gle Liang , 1987). U ser com m unica t ion wi thth i s robo t ha s been r e s t r i c t ed t o one -word commands t ha t co r r e spond to p rede f inedpr imi t ive ac t ions . W ork un der w ay wi ll a l low the phys ica l ly d i sab led u ser to teachth i s robo t how to pe r fo rm complex t a sks i n t he home and a t t he workp lace u s ing

o rd ina ry Eng l i sh commands such a sGo to the table while avoiding the chairand Pickup the p l a te a nd pu t it i n t he mic row ave oven .So m e of ou r d i scuss ion w i ll re f lec t thi sr ecen t work .

I t i s u se fu l t o con t r a s t t he work r epo r t ed i n t h i s pape r w i th VanLehn ' s s t udy o fthe acquis i tion of p ro ced ura l sk il l in m athem at ics (V anL ehn , 1983). V anL ehndiscusses severa l form s of m achine learn ing: l ea rn ing by be ing to ld , l ea rn ing bydiscov ery, lea rn ing b y ana log y, and learn ing by genera l iz ing e xam ples . H e focuses h iswork on the l a s t c a t ego ry, po in t ing ou t t ha t a r i t hme t i c t ex tbooks con ta in manyexe rc is e s and w ork ed exa mp les bu t f ew p rec i se na tu ra l- l anguage p rocedura ldesc r ip t i ons. I n t e rms o f Van Leh n ' s ca t ego r i e s , ou r work li es som ew here be tw eenlea rn ing by be ing to ld and l ea rn ing f rom exam ples . W e exam ine the k ind o f l e arn ingtha t t akes p lace when ins t ruc t ion i s g ivenin terms o f specific exam plesa m e t h o dwe l l su i ted t o t he k ind o f one -on -o ne in st ruct ion we env i sage fo r robo t s . The use o fspec if ic exa m ples a l lows the inev i tab le am bigui t ies of na tura l l anguage to bereso lved in the course of the ins t ruc t ion .

This paper i s o rganized as fo l lows . Sec t ion 2 d i scusses the par t icu la r use we makeof t he m ode l - theo re t i c app roac h to s eman t ic s . I t t hen de f ines t he s e t M o f mode l sfor the a r i thmet ic ins t ruc t ion contex t . In a typ ica l ins t ruc t ion sess ion the agent i spresen ted wi th an exerc i se and g iven verba l ins t ruc t ion based on tha t exerc i se . An

7

3n s w e r r o w

F IG . 1 . A s i n g l e c o l u m n a d d i t i o n e x e r c i s e .



CONTEXT FIXING SEMANTICS 373

example i s t he fo l l owing sequence o f commands fo r s i ng l e - co lumn add i t i on and theexercise in Fig. 1 .

L o o k a t t h e t o p n u m b e r.R e m e m b e r t h e n u m b e r.L o o k a t t he n e x t n u m b e r d o w n .A d d t he tw o n u m b e r s .R e m e m b e r t h e s u m .L o o k a t t h e n e x t s p o t d o w n .I f y o u s e e a n u m b e r c o n t in u e ad d in g r e m e m b e r i n g a n d l o o k i n g d o w n u n ti l y o u s e ea bar.L o o k o n e s p o t d o w n .W ri te the s ing les d ig i t o f the answ er.

Look one spo t l e f t .Wr i te the t ens d ig i t o f the answer.

The se t M o f mod e l s i den ti fi e s a s pe rcep tua l o b j ec t s t he d ig i ts , ba r s , and em ptyspaces of an a r i thme t ic ex erc i se . I t a lso def ines the re la t ionsho r i zon t a l l y l e ft o fandver t i ca l ly be lowt ha t ho ld be tween pe rcep tua l ob j ec t s . The cogn i t i ve and pe rcep tua lfunc t ion ing of the agent be ing ins t ruc ted i s charac te r ized by a reg is te r-machinemod e l . T he re is a s t imu lus - suppor t ed r eg is t e r (SS) wh ich ho lds an image o f t heperc eptu a l ob jec t tha t is a t the agen t s po in t o f v i sua l focus . Then there i s anon- s t imu lus - suppor t ed r eg i st e r, wh ich is know n a s t he me m ory r eg i s te r (M R) . A n

ope ra t ion reg is te r (OP ) ho lds the resu l t o f an addi t ion ope ra t ion . A f ina l reg is te r(ORD) iden t i f i e s an o rde r ing r e l a t i on on pe rcep tua l ob j ec t s .

Sec t ion 3 shows how the deno ta t i ons o f ind iv idua l w ords a r e exp re s sed in t e rms o fthe mode l s de f ined in s ec t i on 2 . I t a l so shows how these deno ta t i ons en t e r i n to t hesyntac t ic ana lys i s o f a sen tence . I t in t roduc es ou r repr esen ta t ion of lex ica l i t ems andexp la in s the u se we m ake o f g ramm ars . Th i s s ec ti on a lso d is cus se s som e imp lem en-tat ion issues .

Sec t ion 4 p re sen t s s eve ra l ex t en ded exam ples o f com m and in t e rp re ta t i on . O fpar t icu la r im por ta nce in th is sec t ion is the ac coun t of how the in te rpre ta t ion of aw ord i s fixed on the ac tua l o ccas ion o f it s use , tha t i s, as par t o f a par t icu la rcommand a t a par t icu la r po in t in the ins t ruc t ion . This sec t ion shows how res idua lcon tex tua l f ac to r s in a word a r e r e so lved by the p e rcep tua l s i tua t ion and the word sp lace in the d i scourse .

The f ina l sec t ion , 5 , po in ts to our cur ren t and fu ture work on ins t ruc tab le robots .An Ap pend ix co n ta in s a g lo s sa ry o f t e rms and n o ta t i on ap pea r ing in the pape r.

2 T h e s e t o f m o d e l s

2.1. WHAT KIND OF MODELS

Ou r app roac h to the i n t e rp re t a ti on o f na tu ra l- l anguage co m m and s st r add le s twot rad i tions : m ode l - theo re t ic sem ant ics from log ic and ope ra t iona l sem ant ics asd e v e l o p e d f o r p ro g r a m m i n g l a ng u a g es . W e b e g i n b y ex p la in in g h o w o u r w o r kex tends t he u sua l mode l - theo re t i c appa ra tu s t o encompass ope ra t i ona l s eman t i c s fo rna tu ra l - l anguage d i scou r se .



74 c CRANG LE AND P SUPPES

As stated earlier, all commands to the agent are interpreted relative to a set ofmodels that characterize the agent to whom instruction is being given and theperceptual sit uation in which the instruction is taking place. The command Find the

empty space to the arithmetic robot of Maas and Suppes, for instance, is interpretedrelative to the rows and columns of an arithmetic problem. That same commandgiven to the mobile base of the robotic aid would be interpreted relative to theconfiguration of objects and their parts in the room in which the instruction is takingplace.

One natural outco me of this viewpoint is that the set of logically possible modelssatisfying a given utterance or piece of discourse is not of primary interest. For allnatural discourse we consider, it is appropriate rather to take a subset of the set ofpossible models. We restrict the set of models to just those that encompass aparticular environment. For example, for the arithmetic instruction the models arerestricted to the ar rangement of digits, bars, and spaces that constitute an arithmeticexercise. For the robotic aid, the models are restricted to the room and its physicalcontents. The aim is to capture the notion of possibility that lies behind ordinarydiscourse.

At the same time as we restrict ourselves to a fixed set of models, we also extendthe models to include a framework for the cognitive, perceptual, and motorfunctioning of the agent to whom the commands are addressed. This means that ifwe are talking about the robotic aid we are not simply restricting ourselves to thephysical objects in the room but must have a way of dealing in the models with the

cognitive and perceptual states of the robot. The language of communication will, aswe envisage it, be almost entirely physical in character. In a command like Go to thetable and pour me a glass of water all of the terms have a direct physicalinterpretation. But interpreting that command relative to a set of models requiressome apparatus to express as part of the model the cognitive state and perceptualactivity of the robot. This point has special plausibility for verbs such as rememberand look at in command s such as Remember where you placed the cup or Look at thenext number. In the work reported here, such verbs are expressed in terms of aregister-machine model that characterizes t he agent s cognitive and perceptualfunctioning. For the mobile robotic aid, moto r functioning is also included.

By taking into account the cognitive and perceptual functioning of the agent, weshare in the tradition of operational semantics which asserts that characteristics ofthe interpreting machine (suitably abstracted) are an integral part of the meaning ofa programming language. As sections 3 and 4 will make clear, however, there aresignificant differences between operational semantics as it has been proposed forprogramming languages and the semantics we propose for natural languages, notleast of which is the utter reliance of natural-language interpretation on the contextat the time of utterance. Nonetheless, we emphasize the procedural basis of oursemantics in that it has a distinct advantage over other, nonprocedural semantics for

natural languages. The procedures provide a direct link between language andaction, that is, between verbal commands and responses to them. This link betweenlanguage and action is especially important in the instructable robot project wherewe want the robot not merely to understand the commands but to take specific andappropriate action in response.

Our use of a set of models to define the context of an utterance has something in



CONTEXT-FIXING SEMANTICS 375

c o m m o n w i th t h e c o m m o n s e n s e m e t a p h y si c s a p p r o a c h t o t he le x ic o n ( H o b b set a l .1986) i n wh ich co re t heo r i e s a r e cons t ruc t ed abou t phys i ca l ob j ec t s - - and abou tt ime, spa ce , m ater ia l and so on- -w i th the lex ica l i tems be ing charac te rized in te rms

of those t heo r i e s . O ur wo rk is d i f fe r en t , howe ve r, i n tha t we focus r a the r m orec lose ly on spec i f ic contex ts o f use , tha t i s , the ac tua l phys ica l envi ronments a t thet ime of the u t te ran ce , and w e mak e prov is ion in the lex ica l i t ems for those contex ts ,a t time o f u t te r ance , t o ma ke the i r con t ribu t ion .

2.2. THE PERC EPTUA L SITUATION

In examin ing the l anguage used to i n s t ruc t an agen t i n e l emen ta ry ma thema t i c s , wefocus on the a lgo r i thms ch i ld ren a r e com m only t augh t fo r co lumn add i ti on ,subt rac t ion , m ul t ip l ica t ion , and d iv is ion . W e assume tha t a ll exerc i ses a re prese n tedin s tandard form at , w i th eno ugh regular i ty in the a r ran gem ent of the f igures tha t theagen t can make a j udgmen t abou t wha t coun t s a s a b l ank space and wha t mere lysepa ra t e s one sym bo l f rom an o the r. A two-co lum n add i ti on exe rc is e , fo r exam ple ,may appea r a s shown in F ig . 2 .

W hi le w e g ive no acco un t o f t he p roces s ing o f v isual i n fo rma t ion f rom thea r i t hme t i c exe rc i s e s , we t ake pe rcep tua l func t ion ing in to accoun t i n t ha t we makeexpl ic i t the perc eptu a l assum pt ions tha t l ie beh ind the use of l anguage in the g ivenpe rcep tua l s i tua t ion . These a s sumpt ions a r e exp re s sed , i n pa rt , a s ax ioms tha t de f inea s imp le pe rcep tua l geo m e t ry o f ob j ec t s , A l though the de t a il s o f t hi s geom e t ry a r es t ra igh t fo rward , t hey a r e im por t an t i n t ha t t he ob j ec t s o f pe rcep t ion they iden t i fy - -nam e ly, t he d ig it s, ba r s , and b l ank spaces o f an a r i thme t i c e xe rc i s e - - a r e t he ob j ec t sre fe r red to in the na tura l - language ins t ruc t ion . In teaching someone to se t the tab le ,fo r example , we do no t cus tomar i ly r e f e r t o edges , su r f aces , and p l anes , bu t t ope rcep tua l ly g iven ob jec t s such a s cups , spoons , and t ab l e s . Ou r t r ea tmen t o fpe rcep t ion is t hus a t t he l eve l app rop r i a t e t o t he dem ands o f na tu ra l- l anguagesemant ics in tha t i t dea ls wi th percept ion a t the leve l o f na tura l - language d iscourse .

We fee l i t i s a p roduct ive s t ra tegy to keep language process ing d is t inc t f rom thepe rcep tua l p roces se s by wh ich edges , su r f aces , and shapes a r e t r ans fo rm ed in tope rcep tua l ly r ecogn ized ob jec t s . Bu t a s t he pape r w i l l make c l ea r, pe rcep tua l

func t ion ing i s in tegra l to our approach for i t i s on ly in in te rac t ion wi th thepe rcep tua l env i ronm en t t ha t t he agen t can in t e rp re t m any na tu ra l -l anguage com -mands , The agen t m us t , f o r exam ple , no t on ly recogn ize ob j ec t s bu t pe rce ive spa ti alr e l a t i ons t ha t ho ld be tween ob jec t s - - r e l a t i ons such a st o t h e l e f t o fand ver t i ca l lyb e l o w - - i n order to in te rpre t ce r ta in words .

W e a s sum e tha t t he agen t a t t ends pe rce p tua l ly to one sym bo l o f t he a r i thme t i cexerc i se a t a t ime . Th e agent s p o in t o f v i sua l focus chang es by m oving ver t ica lly orhorizon tal ly. I ts poin t of visual focus wil l cha nge , f or instance, as a resul t ofunde r s t and ing and o bey ing a com m and such a sL o o k a t t he n e x t s p o t d o w no r L o o kd o w n u n t i l y o u s e e a ba r.A spot in the contex t o f the a r i thmet ic exerc i ses i s the a rea

7

4 8

96 6

l~G. 2. A two-column addition exercise used in instruction.



376 C C R A N G L E A N D P S U P P E S

i m m e d i a t e l y s u r r o u n d i n g , a n d c o n t a i n i n g , a t m o s t o n e s y m b o l . A n o p e n s p o tcon t a in s ne i t he r a d ig i t no r t he ba r cha r ac t e r. I n t he exe r c i s e i n F ig . 2 t he r e i s a no p e n s p o t t o t h e l e f t o f th e 9 a n d b e l o w t h e 4 , a n d t h e r e a r e t w o o p e n s p o ts i n t h e

a n s w e r r o w.T h e a g e n t s v i s u a l g e o m e t r y i s o rg a n i z e d b y a s e t o f a x i o m s . W e i d e n t i f y t w or e l a t i o n s , H a n d V, w h i c h i n t h e i n t e n d e d i n t e r p r e t a t i o n s a r e t h e r e l a t i o n shor i zon ta l ly l e f t o fa n d ver t ica l ly be low.T h a t i s , f o r a n y t w o o b j e c t s a a n db a H b i fand on ly i f a i s ho r i z on t a l l y l e f t o f b , anda V b i f an d on ly i f a i s ve r t i ca l ly be low b . Inthe a r i t hm e t i c exe r c i s e i n F ig . 2 , t he fo l l owing a r e a l l v e r t i c a l ly be low 7 : 8 , 9 , 6 ( t her i g h t m o s t o n e ) , a n d t h e b a r c h a r a c t e r ( t h e r i g h t m o s t o n e ) . F u r t h e r m o r e , 1 isho r i zon t a l l y l e f t o f 7 , 4 i s ho r i zon t a l l y l e f t o f 8 , and so on . No te t ha t 4 i s ne i t h e rv e r ti c a ll y b e l o w 7 n o r h o r i z o n t a l l y le f t o f 7 in t h e i n t e n d e d i n t e r p r e t a t i o n s . T h ere l a t i ons H and V a r e bo th i r r e f l ex i ve , s t r i c t pa r t i a l o rde r i ngs , t ha t i s , i n add i t i on t ob e i n g i r re f le x i v e t h e y a r e b o t h t r a n s it i v e a n d a s y m m e t r i c .

Th e s e t t ha t H and V pa r t i a l l y o rd e r i s t he s e t o f pe r c ep tua l ob j ec t s i n a g iv e na r i t h m e t i c e x e r c is e . W e u s e t h e n m e m o n i c P O B J f o r t h is se t . F o r t h e e x e r c is e in F i g.2 , t he s e t PO BJ con t a in s pe r c ep tu a l o b j ec t s 1 , 7 , 4 , 8 , 9 , 6 , 6 ( bo th oc cu r r en ces o f 6 ),

, ( a g a i n b o t h o c c u r r e n c e s o f t h e b a r c h a r a c t e r ) , a n d se v e r a l b l a n k s p a c e s ( o n eb e l o w t h e 4 a n d t o t h e l e f t o f t h e 9 , o n e b e l o w e a c h o f th e b a r c h a r a c t e r s , a t le a s to n e t o t h e l e f t o f 1 , a n d s o o n ) . N o t e t h a t t h e e l e m e n t s o f P O B J a r e th e s e l o c a t e ds y m b o l s . T o u s e a f a m i l i a r d i s t in c t i o n , th e s e l o c a t e d s y m b o l s a r e t o k e n s , n o t t y p e s .F o r e x p o s i t o r y p u r p o s e s w e r e f e r t o th e s e l o c a t e d s y m b o l s w h e n n e c e s s a r y b y

subsc r i p t i ng t hem a s if t he y we re l oca t ed i n a g r id wh ose o r i g in we re a t t he d ig i t i nt h e t o p r i g h t m o s t c o r n e r w i t h th e X c o o r d i n a t e g i v in g t h e h o r i z o n t a l p o s i ti o n( n u m b e r e d f r o m r i g h t t o le f t) a n d t h e Y c o o r d i n a t e t h e v e rt ic a l ( n u m b e r e d f r o m t o pt o b o t t o m ) . T h u s w e r e f e r t o t h e p e r c e p t u a l o b j e c t s 7 H , 81 2, - ~ , a n d s o o n . T h e s e tP O B J is a l w a y s f in i te a n d f o r t h e s t a n d a r d a r i t h m e t i c e x e r c is e s w e c a n l im i t P O B J t oi n c l u d e o n l y t h o s e b l a n k s p a c e s t h a t a r e i n t h e a n s w e r r o w, t h o s e t h a t a r e w i t h in th ec o l u m n s o f f i g u r e s , a n d t h o s e i m m e d i a t e l y t o t h e l e f t o f t h e l e f t m o s t c o l u m n o ff i gu re s . We use t he symbo l o , subsc r i p t ed a s f o r t he d ig i t s and ba r s , t o r e f e r t o t he sel o c a t e d b l a n k s p a c e s. E a c h t i m e a s y m b o l is e r a s e d o r a n e w s y m b o l is w r i t t e n i n a na r i t h m e t i c e x e r c i s e ( a s w h e n t h e a g e n t w r i t e s t h e a n s w e r i n t h e a n s w e r r o w ) , w eh a v e a n e w s t r u c t u r e P O B J , w h i c h i n m o d e l - t h e o r e t i c t e r m s y i e l d s a n e w m o d e l f o rt he s e t o f ax ioms .

W e m u s t i d e n t i f y t h r e e s u b s e t s o f P O B J f o r th e n a t u r a l - l a n g u a g e i n s t r u c t io n . F i rs tt h e r e i s B L N K , t h e s u b s e t o f l o c a t e d b la n k s p a c e s , t h e n B A R , t h e s u b s e t o fl o c a t e d b a r c h a r a c t e r s , a n d f i n a l ly D I G I T , t h e s u b s e t o f l o c a te d d i gi ts . F o r t h ee x e r c is e g i v e n e a r l ie r , f o r i n s t a n c e , B A R = { -~ 5 ,- 25 } a n d D I G I T ={711,8x2 ,913 ,614 , 121 ,422 ,624}. T he se subs e t s com ple te ly pa r t i t io n P O B J an d ide n t i fyt h e t h r e e t y p e s o f p e r c e p t u a l o b j e c t s o c c u r r i n g in t h e i n s t r u c t io n c o n t e x t . I n a d d i ti o nt o t h e s e p e r c e p t u a l o b j e c t s , o t h e r o b j e c t s m u s t b e i d e n t if i e d f o r th e a r i t h m e t i c

i n s t r u c ti o n , s p e c if ic a ll y t h o s e t h a t r e s u lt f r o m a n a d d i t i o n o p e r a t i o n p e r f o r m e d b ythe agen t . Be fo re i n s t ruc t i on s t a r t s , i t i s a s sumed t ha t t he agen t i s a l r e ady ab l e t oadd any s i ng l e d ig i t t o any o the r d ig i t o r d ig i t pa i r. (Fo r s imp l i c i t y, we l im i tou r se lve s t o d ig it pa i r s . ) T he r e su l t o f such an ad d i t i on , a l so a t oken , i s r e f e r r ed t oa s t he s u m o r t h e a n s w e rdu r ing i n s t ruc t i on .

To de sc r i be t h e age n t s pe r ce p tua l f unc t i on ing in mo re de t a i l , i t i s u se fu l t o de f in et w o o t h e r r e l a t i o n s w h i c h i n t h e i n t e n d e d i n t e r p r e t a t i o n a r e t h e r e l a t i o n s



CONTEXT-FIXING SEMANTICS 37 7

horizontally r ight ofa n d vertically above.F o r a n y t w o p e r c e p t u a l o b j e c t s a a n db, ais h o r i z o n t a l l y r i g h t o f b i f a n d o n l y i f b i s h o r i z o n t a l l y l e f t o f a , a n d a i s v e r t i c a l l ya b o v e b i f a n d o n l y i f b i s v e r t i c a l l y b e l o w a .

T h e f i r s t a x i o m w e i n t r o d u c e i s t h e f o l l o w i n g : F o r a l l p e r c e p t u a l o b j e c t s a a n d b , i fa s t a n d s i n t h e H r e l a t i o n t o b t h e n a d o e s n o t s t a n d i n t h e V r e l a t i o n t o b a n d bd o e s n o t s t a n d i n t h e V r e l a t i o n t o a , a n d s i m i l a r ly f o r V. T h i s a x i o m s i m p l ys t ip u l a te s t h a t a p e r c e p t u a l o b j e c t is n e v e r b o t h in a h o r i z o n t a l a n d a v e rt i ca lr e la t io n t o a n o t h e r p e r c e p t u a l o b j e c t .

T h e s e c o n d a x i o m e n s u r e s t h a t e a c h p e r c e p t u a l o b j e c t s t a n d s i n a n a p p r o p r i a t er e l a ti o n t o o n e d i s t in g u i s h e d e l e m e n t , n a m e l y t h e t o p r i g h t m o s t d i g it o f th ea r i t h m e t i c e x e r c i s e . T h e a x i o m s a y s t h a t t h e r e i s a d i g i t i n t h e t o p r i g h t m o s t c o r n e ro f a n a r i t h m e t i c e x e r c i s e a n d a l l o t h e r d i g i ts , s p a c e s , a n d b a r s a r e h o r i z o n t a l l y l e f t o ft h a t d i g it o r v e r t i c a ll y b e l o w i t, o r a r e v e r t i c a ll y b e l o w a n o t h e r p e r c e p t u a l o b j e c tt h a t i s h o r i z o n t a l l y l e f t o f i t. T h i s a x i o m a l so e s t a b l i s h e s t h e e x i s t e n c e o f t h e d i g itt h a t is in t h e t o p r i g h t m o s t c o r n e r o f a n a r i t h m e t i c e x e r c i s e . I n i ts f o r m a l e x p r e s s i o n ,t hi s a x i o m u s e s t h e n o t i o n o f r e l a ti v e p r o d u c t . ( T h e r e l a ti v e p r o d u c t o f H a n d V ( ins y m b o l s : H / V is t h e r e l a t i o n t h a t h o l d s b e t w e e n a a n d b if a n d o n l y i f t h e r e e x i s ts ac s u c h t h a t a H c a n d cVb . T h e a x i o m s a y s t h a t t h e r e is a u n i q u e p e r c e p t u a l o b j e c t as u ch t h a t a i s a l o c a t e d d i gi t a n d e v e r y o t h e r p e r c e p t u a l o b j e c t b s t a n d s i n th er e l a t i o n b H a o r bVa o r b H / Va .

T h e t h i r d a x i o m s t ip u l a t e s t h a t e a c h p e r c e p t u a l o b j e c t h a s i ts o w n s p o t , t h a t i s,t h a t c h a r a c t e r s a r e n o t s u p e r i m p o s e d o n e a c h o t h e r . I n its f o r m a l e x p r e s s i o n , t h is

a x io m u s e s th e n o t i o n o f im m e d i a t e s u c c e s so r a n d i m m e d i a t e p r e d e c e s s o r . ( W e s a yt h a t a i s a V - i m m e d i a t e s u c c e s s o r o f b i f a n d o n l y i fa V b a n d ( V c ) [ i fc V b t h e n ( a = co r cVa ]. S i m i l a rl y, a is a n H - i m m e d i a t e s u c c e s s o r o f b i f a n d o n l y i fa H b a n d ( V c )[if cr ib t h e n ( a = c o rcHa ] . Th ea x i o m i s a s f o l l o w s : E a c h p e r c e p t u a l o b j e c t a h a sa u n i q u e V - i m m e d i a t e s u c c e s s o r , p r o v i d e d i t h a s a V s u c c e s s o r a t al l ( t h a t i s,p r o v i d e d t h e r e e x i s t s a c s u c h t h a tcVa , a n d a u n i q u e H - i m m e d i a t e s u c c e s s o r ,p r o v i d e d i t h a s a n H s u c c e s s o r a t a ll , a n d s i m i l a r ly f o r p r e d e c e s s o r. T h i s a x i o m a l soe n s u r e s t h a t H a n d V h a v e t h e i n t e n d e d i n t e r p r e t a t i o n o fhorizontal ly lef t ofa n dvertically below.I n t h e e x a m p l e i n F ig . 2 , f o r i n s t a n c e , i t is t h i s a x i o m t h a t p r e v e n t s4 f r o m b e i n g v e r t i c a ll y b e l o w 7 a n d 8 f r o m b e i n g v e r t ic a l ly b e l o w 1. F o r s u p p o s e 4w e r e v e r t i c a l l y b e l o w 7 . T h e n s i n c e 8 is v e r t i c a l l y b e l o w 7 a n d 7 h a s a u n i q u eV- i m m e d i a t e s u c c e s s o r , e i t h e r 8 is t h e s a m e p e r c e p t u a l o b j e c t a s 4, o r 4 is v e r t ic a l lyb e l o w 8 o r 8 is v e r t ic a l l y b e l o w 4 . B u t 8 a n d 4 a r e d i st in c t o b j e c t s . F u r t h e r m o r e , b yt h e f i rs t a x i o m , s i n c e 4 is h o r i z o n t a l l y l e f t o f 8, 4 is n o t v e r t i c a l l y b e l o w 8 n o r i s 8v e r t i ca l l y b e l o w 4 .

S e v e r a l f u r t h e r a x i o m s a r e r e q u i r e d t o d e s c r i b e a w e l l - f o r m e d a r i t h m e t i c e x e r c i se .T h e s e a x i o m s , a s w e l l a s th e p r e v i o u s t h r e e , m u s t h o l d a t e a c h s te p o f t h e e x e r c i s e ,w h e t h e r , f o r i n s ta n c e , t h e r e a r e o n l y b l a n k s p a c e s i n t h e a n s w e r r o w o r s o m e d i g it so f t h e a n s w e r h a v e a l r e a d y b e e n w r i t t e n o u t . T h e s e f u r t h e r a x i o m s a r e l is te d in

Ta b l e 1. To g e t h e r t h e n i n e a x i o m s d e s c r i b e t h e p e r c e p t u a l s i tu a t i o n s in w h ic hi n s t r u c t i o n t a k e s p l a c e .

2.3. THE AGENT S COGNITI VE AND PERCE PTUA L FUNCTIONING

T h e r e a r e s e v e r a l a s p e c t s o f t h e a g e n t s c o g n i t i v e a n d p e r c e p t u a l f u n c t i o n i n g t h a tm u s t b e t a k e n i n t o a c c o u n t f o r t h e s e m a n t i c s . T h e f o l l o w i n g a r e si g n if ic a n t: t h e



78 C CRA NGL E AND P SUPPES

TABLE 1The pe rcep tua l ax ioms

A1 A pe rcep tua l ob jec t is never bo th ho r i zon ta lly l e f t o f and ve r ti ca lly be low or abovea n o t h e r p e r c e p tu a l o b j e c t

A2 Th ere i s a d ig i t in the top r ightm ost corne r of an ar i thmet ic exerc ise and a l l o th er d ig it s,spaces , and bars are hor izon ta l ly lef t of tha t d ig i t or ver t ica l ly below i t , or are ver t ica l lybe low a no th e r p e rcep tua l ob jec t tha t is ho r i zon ta lly le f t o f it

A3 Each ob jec t has i ts own spo t , tha t i s, cha rac te r s a re no t super imposed on each o the r

A4 No b lank space appear s in the r igh tmos t co lumn o f f igu res above the ba r

A5 Th ere are a t leas t two digi ts ver t ica l ly abov e the r ightm ost ba r

A6 Each co lumn con ta ins exac t ly one ba r

A7 The ba r s a re a ll ho r i zon ta l ly ad jacen t

AS Only one d ig i t o r b l ank space appear s be low the ba r in each co lumn

A 9 No digi t i s hor izo nta l ly to the lef t of a b lank space

a g e n t s v i s u a l p e r c e p t i o n , t h e b a s i c a r i t h m e t i c s k il ls th e a g e n t s t a r ts w i t h , t h e a g e n t s

m e m o r y c a p a b i li ti e s , a n d t h e a g e n t s a b il it y t o w r i t e d o w n s y m b o l s f o r t h e a n s w e r.I n d e f in i n g t h e c la ss o f m e n t a l m o d e l s a p p r o p r i a t e t o t h e a r i t h m e t i c i n st r u c ti o n ,

w e u s e t h e i d e a , f a m i l ia r f r o m a u t o m a t o n t h e o r y, o f a r e g i s t e r a n d i ts c o n t e n t . I ng e n e r a l , t h e r e is a s e t o f r e g i st e r s R ~, R 2 . . . . . R n , a n d t w o o p e r a t i o n s o n t h e s er e g is t e rs . O n e o p e r a t i o n y i e ld s t h e c o n t e n t o f a r e g i s te r ; th e o t h e r s e t s t h e c o n t e n t o fa r e g i st e r . W e d e s i g n a t e t h e f i rs t o p e r a t i o n u s i n g s q u a r e b r a c e s a r o u n d t h e r e g i st e rn a m e a n d w i t h a s u b s c r i p t t o i n d i c a t e t i m e . T h a t i s, w e w r i t e [ R i] , t o s h o w t h ec o n t e n t o f r e g i s t e r R , a t t i m e t, w i t h [ R ~], = x m e a n i n g t h a t t h e c o n t e n t o f r e g i s t e r R~a t t i m e t i s x . T h e s e c o n d o p e r a t i o n a s s ig n s c o n t e n t t o a r e g i s t e r a t t i m e t. W e u s et h e a s s i g n m e n t o p e r a t o r *--,. T h u sRi -- ,x m e a n s t h a t t h e c o n t e n t o f r e g i s t e r R i is s e tt o x a t t i m e t. F o r o u r p u r p o s e s , w e n e e d o n l y id e n t i f y t h e t i m e t b e f o r e a p r o c e d u r eis e x e c u t e d , t h e t i m e t + 1 o f i ts e x e c u t i o n , a n d , w h e r e n e c e s s a r y, t h e t i m e t + 2 a f t e ri t s e x e c u t i o n . I f R , * - - , x t h e n [R i],+ ~ = x .

W h a t d o e s x r a n g e o v e r ? T h e c o n t e n t o f a r e g i s t e r is e i t h e r a n i m a g e o f ap e r c e p t u a l o b j e c t o r a n o b j e c t t h a t r es u l ts f r o m th e a d d i t i o n o p e r a t i o n . F o re x a m p l e , t h e i m a g e o f p e r c e p t u a l o b j e c t 723 in o n e r e g i s te r is a d d e d t o t h e i m a g e o ft h e p e r c e p t u a l o b j e c t 4 22 t h a t is th e c o n t e n t o f a n o t h e r r e g i s t e r to y i e l d t h e d i g it p ai r11. F o r s i m p l i c i t y o f l a n g u a g e , w e r e f e r t o d i g i ts ( a n d d i g i t p a ir s s u c h a s 11 ) t h a t a r en o t i m a g e s o f p e r c e p t u a l o b j e c t s a s m e n t a l o b j e c t s a n d w e d o n o t c a ll i m a g e s m e n t a l

o b j e c t s .F o r t h e a r i t h m e t i c i n s t r u c t i o n , w e p r o p o s e a m i n i m a l c o g n i t iv e a r c h i t e c t u r e b y

s p e c i f y i n g t h e f o l l o w i n g s e t o f f o u r r e g i s t e rs . T h i s c h o i c e o f r e g i s t e r s e t f o l lo w s t h a tm a d e b y S u p p e s ( 1 9 73 b ; S u p p e set al . , 1 9 8 3 ) i n a d i f f e r e n t b u t r e l a t e d s t u d y o fa r i t h m e t i c p e r f o r m a n c e , w i t h t h e a d d i t i o n o f o n e r e g i s t e r t h a t p l a y s a s p e ci a l ro l e inf ix i n g c o n t e x t . A s w e w i ll d is c u ss b e l o w , t h e c h o i c e o f r e g i s t e r s h a s a d i r e c t e f f e c t o n




TABLE 2The role of the registers

SS:

M R :

OP:

O R D :

A s t imulus-supp or ted regis ter. I t holds an image of the perc eptual objec t tha t i s a tthe agen t s po int of v isual focus .

A non-s t imulus - suppor ted reg i s te r, r e fe r red to a s the m em ory reg i st e r. The con ten tof th is regis ter i s e i ther an im age, t ran sfer red f ro m SS, or a me nta l objec t ,t r a n sf e r r e d f r o m O P.

A n o pera t ion regis ter. This regis ter holds the resul t of an addi t ion oper a t ion , amen ta l ob jec t .

An ord er ing regis ter. This regis ter ident if ies an ord er ing re la t ion on percep tualobjec ts .

w h a t c o m m a n d s o r s e q u e n c e s o f c o m m a n d s c a n b e o b e y e d b y t h e a g en t . T h e f o u rr e g i st e r s a r e d e s c r i b e d i n Ta b l e 2 .

T h e r e a r e f i v e a c t i v it i e s i n t h e a r i t h m e t i c i n s t r u c t i o n c o n t e x t : l o o k i n g , s e e in g ,r e m e m b e r i n g , w r i t in g , a n d a d d i n g . T h e r e g i st e r s a r e l in k e d t o t h e a c ti v it ie s i n t h ef o l l o w i n g w a y . F i r s t , e a c h a c t i v it y i s c h a r a c t e r i z e d a s a t r a n s f e r b e t w e e n r e g i s t e rs ,o f t e n a s s e v e r a l a l t e r n a t i v e t r a n s f e r s . S e c o n d , e a c h a c t iv i t y is a l so c h a r a c t e r i z e d int e r m s o f o p e r a t i o n s p e r f o r m e d o n t h e c o n t e n t o f t h e s o u r c e r e g is te r . T h e n a r r a t i v ed e s c r i p t i o n t h a t f o l l o w s e l a b o r a t e s o n t h e s e t r a n s f e rs a n d s h o w s e x a c t l y w h a t

o p e r a t i o n s a r e p e r f o r m e d o n t h e s o u r c e r e g i st e r c o n t e n t b e f o r e t h e t ra n s fe r . T h er e g i s te r t r a n s f e r s f o r e a c h a c t iv i ty a r e s u m m a r i z e d i n Ta b l e 3 .

Looking W e i d e n t i fy t w o f o r m s o f lo o k i n g i n o u r i n s t r u c t io n c o n t e x t . T h e f i rs ti s a n o b j e c t - o r i e n t e d a c t i v i t y, r e f e r r e d t o i n s e n t e n c e s s u c h a sLook at the topnumber o r Look at the next spot T h i s a c t i v i ty h a s a p e r c e p t u a l o b j e c t a s it st a r g e t - - t h e t o p n u m b e r o r t h e n e x t s p o t , f o r in s ta n c e. T h e s e c o n d f o r m o f l o o k i n g isr e f e r r e d t o i n s e n t e n c e s s u c h a sLook three spots down o r Look down until you see a

TABLE 3

The register transfers

Sourc e regis ter De st ina t ion regis ter

Lo oking SS SSSeeing SS SS

R e m e m b e r i n g S S M RO P M R

W ri ting M R SS

SS SSO P S S

A d d i n g S S a n d M R O PS S a n d O P O P

M R a n d O P O P




bar. It is characterized by the fact that the looking is in a certain direction and thatthere is a condition for when to stop looking----either a distance that must be covered(three spots) or a state of affairs that must hold (you see a bar). The effect of both

these forms of looking is that the content of SS changes. The image of theperceptual object in SS before the command is obeyed is replaced by the image ofanother perceptual object, the one at the new point of visual focus. When thelooking is in a certain direction, the ordering relation in ORD is used to constrainwhich perceptual objects can be looked at.

Seeing. Seeing takes into account only the con tent of SS. It does not change thatcontent but checks what it is. References to seeing typically appear in sentences suchas You see a bar and You see a number which in our instruction context areembedded in imperatives such as Look down until you see a bar or If you see anumber, add the number to the sum.

Remembering. Remembering transfers to MR either the image of a perceptualobject or a mental object that is a digit or pair of digits, the result of an earlieraddition operation. Typical remember-commands are Remember the sum andRemember the number. A command such as Remember the number would of courseoccur after a perceptual object had been identified through a command such asLook at the top number. It is clear that the transfer to MR could originate in eitherSS Remember the number) or OP Remember the sum).

Writing. The activity of writing goes beyond simple register tranfers andoperations on register content. It causes a new perceptual object to come into being,

replacing a blank space in the answer row with that new object. Typicalwrite-commands are: Write the singles digit of the answer, Write the answer in thenext spot open, Write the number you see, and Write the number you remembered. Itis clear that the object to be written out may come from memory (the content ofMR), may be specified perceptually (the content of SS), or may be the result of anarithmetic operation (the content of OP). Writing is thus an action that takes anobject (the content of SS or MR or OP) and transforms it into a new perceptualobject. After writing out a character, the agent continues to attend to thatperceptual object. Three conditions govern the action of writing: only single digitsmay be written, only blank spaces below the bar may be written over, and a symbolis written out at the agent s point o f visual focus (the agent does not look elsewherewhile writing). Note that in writing over a blank space, the axiom stating that eachperceptual object has its own spot still holds because the new perceptual object andthe old are not from the same structure. If a is the perceptual object being replacedin POBJ, the old structure, and b is the new perceptual object, then it is the casethat PO B J - {a} = P O B J - { b } where POBJ is the new structure. An operationmust be defined for the semantics of write that will produce the new perceptualobject at the location of a. This operation is defined over the models themselves; itis not a primitive of the perceptual geomet ry, nor of the agent s cognitive and

perceptual architecture. If the relationship between successive perceptual modelswere itself given systematic treat ment, it would of course be part of that theory. Butbecause we have only one world-changing activ ity- -the activity of writing --we willleave the discussion in its present informal state.

Adding. Adding is a cognitive and perceptual activity that forms the arithmeticsum of two digits or digit pairs. We assume that before all instruction starts, the



CONTEXT FIXING SEMANTICS 38 1

agent is able to add any digi t to any other digi t or digi t pair. These digi ts are givenpe rcep tua l ly o r a r e i n memory. The add i t i on ope ra t i on i s t he re fo re de f ined ove rbo th t he s e t o f men ta l ob j ec t s and the s e t o f images o f pe rcep tua l ob j ec t s . Typ ica l

add-commands are: A d d the two numbers Ad d the number to the sum Ad d thenumber you remembered to the sum and Add the number you see to the sum. It isc l ea r f rom these s en t enc es t ha t t he add i ti on op e ran ds cou ld come f rom any two o fSS , M R , and O P. Be cau se t he re i s on ly one s t imu lus - suppor t ed r eg i st e r, SS , a t l e a stone ope ra nd mus t be i n mem ory, i n e i t he r M R o r OP . Th e fo l l owing in st ruc t ionsequence t he re fo re can no t be i n t e rp re t ed by ou r agen t :

Look at the top number.Look at the next number down.Add the two numbers.

The fo l lowing sequence , on t he o the r hand , can be g iven an i n t e rp re t a t i on :

Look at the top number.Remember the number.Loo k at the next number down.Add the two numbers.

Perhap s t he m os t s t ri k ing f ea tu re o f t he cogn i ti ve and pe rcep tua l a r ch i tec tu re wehave prop ose d i s i ts s impl ic i ty. Th ere i s very l im i ted cen t ra l p roce ss ing capabi l i ty :fou r r eg i s t e r s and two bas i c ope ra t i ons t o acces s and change the con ten t o f a

reg is te r. As sec t ions 3 and 4 wi l l revea l , the computa t iona l complexi ty l i es in thelex ica l i t ems themselves and in the ru les of the grammar.

The p reced ing ax ioms and de f in i t i ons and the r eg i s t e r appa ra tu s come toge the r t odef ine the c lass of mo dels in te rms o f which the langua ge is in te rpre ted . The semodels subs tan t ia l ly res t r ic t the phys ica l envi ronments under cons idera t ion to jus tt hose in wh ich the l anguage o f a ri t hme t ic i n s truc tion m akes s ense , a s add re s sed toan agen t w i th t he k ind o f cogn i ti ve and pe rce p tua l func tion ing desc r ibed . T hefu r the r spec i a l pu rpose s e rved by th i s c l a s s o f mode l s , and one we wi l l e l abo ra t e i nthe next sec t ion , i s tha t i t a l lows the sem ant ic in te rpre ta t ion of the lex ica l i t ems tobe fixed, a t the t ime o f the i r use , by the pe rcep tua l s i tua t ion in which the w ords a reused and b y the i r p lac e in the d i scou rse . Tw o exam ples i l lus t ra te th i s idea br ie f lybefore i t i s t aken up in some de ta i l in the next sec t ion .

The in t e rp re t a t ion o f t he wo rdtop as u sed in a command such a sLook at the topnumber o r Fetch the cup from the top shelf will de pe nd on the organiza t ion of thepe rcep tua l ob j ec t s - - -number s o r she lves - - - i n t he env i ronmen t . Some sequenc ing o ro rde r ing r e l a t ion w i ll be e v id en t - - t h e she lves , f o r i n s tance w i ll t yp ica l ly be o rd e redin te rm s of the i r ver t ica l d i s tance f rom the f loor. I t is th is o rder ing re la t ion tha ta l lows the top she l f to be se lec ted . Express ions such asthe top student in the classa lso dep end on an unde r ly ing o rde r ing r e l a ti on fo r the i n t e rp re t a t ion o ftop but inth i s case , as in o thers , the order ing i s no t g iven perceptua l ly. For many uses oftophow eve r, and for a ll uses in con tex ts o f spec ia l in te res t in th i s pap er, the order ingre la t ion i s g iven perceptu a l ly. Thu s in spec i fy ing perc eptu a l re la t ions , the mo delshe lp f ix the in te rpre ta t ion of the words used in ins t ruc t ion .

The r eg i s te r app a ra tu s a l so he lps fix t he i n t e rp re t a t i on o f wo rds in con tex t. F ro mthe summary in Table 3 i t i s c lear tha t the procedure spec i f ica t ions for thewrite



382 c C R A N G L E A N D P S U P P E S

remember a n d add verb s can spec i fy on ly the des t ina t ion reg is te rs , fo r on ly they a reinvar ian t in a l l cases , namely : SS forwrite M R f or remember a n d O P f o radd. Thes o u r c e r eg i st e rs a r e d e t e r m i n e d b y th e c o n t e x t . I f t h e c o m m a n d w e r eRemember the

number f o r in s ta n c e , t h e t ra n s fe r w o u l d b e f r o m S S to M R . I f t h e c o m m a n d w e r eRemember the sum t h e t r a n s f e r w o u l d b e f r o m O P t o M R . Ta k i n g a n e v e n l a rg e rb i te o f c o n te x t , t h e c o m m a n dAdd the numbers would in i t i a te a t ransfer f rom SS andM R , o r S S a n d O P, o r M R a n d O P, d e p e n d i n g o n t h e s e v e r a l c o m m a n d s t h a tpreceded i t . Examples in the fo l lowing sec t ions wi l l show how the lex ica l i t ems a recombined to y i e ld a f i na l r eg i s t e r t r ans fe r fo r t he who le command .

3 G r a m m a r s a n d t h e l e x ic o n

We have adop ted the i dea , f ound in Mi l l e r and Johnson-La i rd (1976) , Suppes (1980 ,1982), C rang le (1984) and o the r s , t ha t n a tu ra l- l anguage u t t e r ances m ay be r ep re -sen t ed s eman t i ca l l y a s p rocedures . Each word deno te s a p rocedure and these l ex i ca lp r o c e d u r e s a r e c o m b i n e d t o f o r m m o r e c o m p l e x p r o c e d u r e s f o r s e n t e n c e s . T h em echan i sm con t ro ll i ng t h is syn thes is o f p roced ures is t he s eman t i c t r ee (Su ppes ,1973a , 1976 , 1979 ; Suppes & Macken 1978) . A s imp le example makes t he concep tclear.

Cons ide r t he fo l l owing pa r se t r ee fo r t he impera t iveFind the empty space. (SeeF ig . 3 . ) The non- t e rmina l l abe ls show n a re I ( fo r impe ra t i ve ) , V P ( fo r ve rb ph ra se ) ,N P ( fo r nou n ph ra se ) , V ( fo r ve rb ) , N ( fo r noun ) , and A D J ( fo r ad j ec t i ve ) . Th i s

pa r se i s p roduced by a con tex t - f r ee g rammar, wh ich we then ex t end by a s s ign ing a tmo s t one s eman t i c func t ion to each p roduc t ion ru le o f t he g ramm ar. (See Tab le 4 .)The seman t i c func t ions s t i pu la t e how the deno ta t i on a t e ach node o f t he t r ee i s t o beob ta ine d f rom the deno ta t i ons o f i ts daugh te r nod es . Th e r e su lt ing g ram m ar i scal led a potentially denoting grammar fo l l owing Suppes (1973a ) . We use squa reb races t o show the deno ta t i ons . Fo r i n s t ance , [NP] s t ands fo r t he den o ta t i on o f NP ,[space] s t ands fo r t he deno t ion o fspace. Two ope ra t i ons appea r i n t he s eman t i cfunc t ions . Th ey a r e des igna t ed b y the sym bo l s * and fq and a r e exp la ined he re . The9 ope ra t ion i s in fac t s imply func t ion appl ica t ion and in the ex am ples la te r in thepaper we wi l l use the usua l paren theses no ta t ion ( tha t i s , we wi l l wr i te [V]( [NP]) ,for ins tance) . The o ther opera t ion i s se t in te rsec t ion . Given our se t - theore t icr ep re sen t a t i on o f p rocedu res ( t o be exp la ined be low) , a l l t he u sua l s e t- t heo re ti cope ra t i ons a r e a t ou r d i sposa l fo r t he s eman t i c func t ions .

I

v P

v P

t h e DJ N

e m p t y s p c eFIG 3. Parse tree.



C O N T E X T F IX I N G S E M A N T I C S 3 8 3

TABLE 4Potent ia l ly deno t ing g ram m ar

Production rule Sem antic function

I---) V P [I] = [VPVP---) V + N P [VPI = [Vl * [NPINP--) the +AD J + N [NP] = [AD J] n [NP]V---~ in d [V] = [find]N ~ space [N] = [space]A D J ~ empty [ADJ] = [empty]

Each lex ica l p rocedure i s g iven a se t - theore t ica l represen ta t ion which in i t ss imples t fo rm appears as fo l lows .

{ x , y ~ : F x ,y)} .

x des igna tes the input ; y des igna tes the ou tput ; and F des igna tes a complexcond i t i on on x and y wh ich e s t ab l i shes t he r e l a t i on be tween the i npu t and ou tpu t ,p lu s any cond i t i ons x has t o mee t t o qua l i fy a s accep tab l e i npu t t o t he p rocedure .These l ex ica l p roce dure s a r e t o be t hough t o f a s mech an i sms fo r t r ansfo rming thes t at e o f t he agen t be ing add re s sed o r t he s t a t e o f t he phys i ca l env i ronm en t . I n t hel ex i ca l p rocedure fo r t he ve rbf ind , for ins tance , x wi l l des igna te the s ta te of the

agen t be fo re af ind-ac t ionhas been t aken and y t he s ta t e o f t he agen t a f t e rwards .C lea r ly, t he se r ep re sen t a t i ons p rov ide spec i f i ca t i ons fo r t he p rocedures , no t t hea lgor ithms them selves . This d i s tinc t ion be tw ee n pro ced ure spec i f ica tions anda lgor i thms wi l l be re turned to la te r in th i s sec t ion when we d iscuss implementa t ion .

To cap tu re t he con tex t -dependency inhe ren t i n many words , t he s eman t i creprese n ta t ion of a w ord i s typ ica l ly g iven not by a s imple inp ut -o utp ut spec i fica tionbu t by an exp re s s ion o f t he fo ll owing fo rm. W e use t he s tanda rd l ambd a no ta t i onfor abs t rac t ion . )

Aocfl){ x, y) :q ~ x, y , o:,fl)}.

W hat we ha ve here se t - theore t ica l ly is s imply a func t ion f rom the se t o f pa i r s{ t~ , f l)} to a se t o f p ro ce du re spec i fica tions . W e le t the var iab le t r range ove rre la tions g iven by the pe rcep tua l s i tua t ion and the var iab le fl range ov er proced uresin t he ir s e t - t heo re t i c r ep re sen t a t i on , p roced ures g iven by the w ords su r round ing thegiven w ord in a sen tence . Th ese ) . -express ions thus prov ide a w ay of le t ting ap rocedure spec i f i ca t i on be de t e rmined in pa r t by t he pe rcep tua l s i t ua t i on i n wh ichthe wo rd is u sed and in pa r t by the o the r w ords su r round ing tha t w ord , two a spec t sof contex t o f spec ia l in te res t in th is paper. The sem ant ic represe n ta t ion of the wordtop , for ins tance , wi l l therefore be a ) . -express ion tha t a l lows the appropr ia te

order ing relat ion to be set as the value of the f i rs t A-variable .Fo r ma ny w ords , t yp i ca ll y nouns and ad j ec t i ve s , t he p rocedures t hey de no te a r e

o I i n te r e s t on ly fo r the i r ou tpu t . The deno ta t i on o fspace, for instance, is aproc edure tha t re turns a ll perc eptu a l o b jec t s tha t a re spaces in the g iven ins truc t ioncon tex t . Our s e t - t heo re t i c r ep re sen t a t i on o f[space] is thus {a :F l a )} wh ere F~s tands for the condi t ions tha t an en t i ty a mus t meet to be d i ffe ren t ia ted f rom o ther



3 8 4 C, CRANGLE AND P. SUPPES

objec ts in the g iven perceptua l s i tua t ion and be ident i f ied as a space . Whi le th isl ex ica l p rocedure does no t i nc lude any th ing o f t he pe rcep tua l p rocesses by wh ichob jec t s a re r ecogn ized , i t does acknowledge an appropr i a t e l eve l o f pe rcep tua l

funct ion ing in tha t i t re turn s the a ppro pr ia te perc eptu a l objec ts . A long s imi lar lines ,t h e d e n o t a t i o n o fempty i s a p roc edu re tha t r e tu rns a l l en t it i e s t ha t a r e em pty and noo the r s . Le t i t s s e t - theo re t i c r ep resen ta t ion be{a:F2 a)}. T hed e n o t a t i o n o ff ind isg iven by the express ion AS){ a , b) :F 3 a , b , S)} w he re a des ignates the input , b theout put , and the va lue o f S is g iven by the im m edia te l inguist ic sur roun d, specif icallyby the den o ta t ion o f t he ob jec t noun phrase , w h ich in ou r exam ple isthe empty space.H ere F3 s tands for the con di t ions tha t es tab l ish th e re la t ion be tw een the input a andthe output b such tha t a i s the objec t tha t i s a t the present poin t of focus , and b , ane lem en t o f S tha t i s , an em pty spo t ) , i s a t t he n ew po in t o f focus . F ina lly, t hedef in i te a r t ic lethe is t r ea t ed synca tegoremat i ca l ly.

Th e ex ten ded g ram m ar y ie lds the sem an t i c t r ee in F ig. 4 fo rFind the em pty space.To the r igh t o f t he co lon a t each nod e we show the den o ta t ion o f t ha t node . A t thetop o f t he t r ee we have a p rocedure spec i f i ca t ion fo r t he commandFind the emptyspace, s ta ted in the impera t ive mood. Whi le th is example i l lus t ra tes the bas ica p p a r a tu s o f t h e s e m a n t i c t re e , s h o w i ng h o w t h e i n p u t - o u t p u t r e p r e se n t a ti o n s o fp rocedures a re composed us ing ope ra t ions such a s func t ion app l i ca t ion and se tin t e r sec t ion to p roduce o the r p rocedure spec i f i ca t ions , s eve ra l ques t ions a reimm edia t e ly r a i sed by the exam ple :

1 . W hat co nten t i s g iven to the cond i t ions F], F2 , and F3 in the lex ica l deno ta t ions?In o the r words , how do w e de f ine the con d i t ions they s t and fo r?2 . W h e r e d o e s t h e i n p u t t o a p r o c e d u r e c o m e f r o m ? I n o t h e r w o r d s, w h e r e d o e s ao f { a , b ) : F a , b)} ge t it s va lue? A nd w ha t happe ns to b , t h e ou tpu t o f t hep r o c e d u r e ?3 . Ho w does a pro ce du re spec i f ica tion , tha t is , an express ion o f the form{ a , b ) : F a , b)} , become an execu tab le ob jec t ? Th i s i s t an tamoun t to a sk ing howthe com plex cond i tion F i s t r ans fo rm ed in to an a lgo r i thm.

l : { ~ a , b ) : F a a , b , { c : F 2 c ) a n d F l c ) } ) }

V P : { a , b ) : F a a , b , { c : F 2 c ) a n d F I c )} )}

V: h S ) { a , b ) : F 3 a ,b , S ) } N P : { a : F 2 a ) a n d F t a ) }

t h e A D d : {a : F2 a ) } N : {a : F 1 a )}

f in d : ? t S){ a , b ) : F 3 a ,b , S ) } e r a p t y : { a : F 2 a ) } s p ac e: { a : F l a ) }

FIG. 4. Sem antic tree.



C O N T E X T F IX I N G S E M A N T I C S 3 8 5

We will answer questions 1 and 2 in this paper. While question 3 is not the topicof this paper, it must of course be answered before an operational system can beproduced, one that allows a specific robot to be instructed in a given context, for the

robot obeys a command by executing the procedure for that command. Only whenthe conditions on input-output are replaced by algorithms that meet thoseconditions do we have something that can be executed.

There are two reasons for putting this question aside for the present. The first isthat an operational system does exist for the arithmetic instruction context (MaasSuppes, 1983, 1985). While the syntactic and semantic basis of that work issomewhat different from the one laid out here, that work stands as a demonstrationof the kind of algorithms needed for arithmetic instruction to be successful. It hasestablished the feasibility of implementing a natural-language system for arithmeticinstruction.

The second reason for putting the question aside in this study is a conceptualrather than a pragmatic one. Natural-language commands seldom entail muchcommitment to how the command is to be obeyed. Take the command Pick up thecup for instance. While there might be some 'normal' or customary way of pickingup a cup---by the handle and keeping the brim level so that the contents do notspill--the command itself does not exclude other ways of picking up the cup. Thislaxity in language is relied on when we qualify verbal commands, by saying, forinstance, Pick up the cup by grasping it at the bottom or Pick up the cup withoutusing the handle. It would thus be a mistake to assume the same algorithmic

interpretation o f pick up for all commands, even in a restricted set of circumstances.There are, however, certain constraints that an action must meet to qualify as apicking-up action in the given instructional context. It is those constraints that mustbe expressed in the lexical entry as the conditions on input and output for thepick-up procedure. Only after all the appropriate contextual features have beenaccounted for can an executable procedure be produced. The concern in this paperis with those contextual matters , particularly those given by the perceptual situation.

However, a few remarks on implementation are appropriate before turning to theanswers to questions 1 and 2. These remarks draw on our recent work with therobotic aid. In moving from input -outpu t conditions to algorithms, the goal is clear:to achieve specific action in response to a verbal command, action that isappropriate in the context. The first step towards this goal is to identify a defaultway of performing each action in the instructional context. For the action ofinserting, for instance, as referred to by the command Put the potato in the potconsider the following sequence of robot acts:

Grasp x.Move the arm (possibly with obstacle avoidance routines active) until x is all the wayinside y.Put x down on a surface in the in terior of y.Withdraw the arm from y.

The problem that lies in wait is that of modify ing this sequence in response to arange of verbal commands that make sense in the given context. In addition tocommands that explicitly modify the actio n--for example, Put the fo rk in the drawerbut do not let go--there are several other examples: Put the cup in the microwave




where we expect the vertical orientation of the cup not to be disturbed; Put thepenci l in the t ray where the pencil must be oriented along its principle axis to fit inthe tray.

As a solution to this problem, we propose the following approach which leadsdirectly from the position taken in this paper. For each word we encode only inpu t-output conditions in the lexical entries, no algorithms. Then, having used the methodsoutlined in this paper to put together an interpretation for the whole command, thedefault procedure for the main verb is modified as stipulated by the conditionscollected from all parts of the command and from the context. On the question ofhow defaults are overridden, one answer (and one we have implemented for themobile base of the robotic aid) is to embed procedures in a highly parallelprocessing environment. The special circumstance that signals the override of adefault will then contribute its own procedure, and the parallel execution ofprocedures will produce the appropriately modified action. There are cases in whichthis approach works and cases in which it does not. Again we draw on the roboticaid for an example. The procedure for go to as used in the commands Go to thetable and Go to the table without hi t t ing the chairproduces straight-line motion ofthe robot towards its goal. The procedure invoked by without hi t t ing the chair onthe other hand, produces motion that avoids the chair. Together, these procedureshave the effect of modifying the robot s direct mov ement towards the table, allowingit to skirt around the edge of the chair. However, not so successful a story can betold for the p i c k u p command. If the default action for picking up a cup has the

robot grasping the handle of the cup, and if wi thout us ing the handlekeeps therobot s gripper away from the handle, the overall effect is that the cup is not graspedat all. For actions such a pick-up and insert, then, the default procedures must bedefined so that all aspects of their operation that are open to modification areparameters that can be set by the context. In terms of the notation alreadyint roduced for the lexical items, 3.-variables must be provided for those aspects ofthe action that may be altered by the context.

We return now to questions 1 and 2 concerning the semantic content of the lexicalitems. The answer to these questions is to be found in the set M of models definedfor the given instruction context. For our example of Find the empty space theinstruction context could be t he ari thmetic exercises we are concerned with here, asin F ind the empty space be low the ba r. Thecommand is equally appropriate,however, in an otfice or at home, two environments in which the robotic aid isexpected to function. In instructing this robot to set the table for a meal, forinstance, the user might tell it to find the empty space for the plate. Alternatively , ifthe robotic aid were being used to do filing, it might be told to find the empty spacein the drawer o f the file cabinet. Quite different models would be required for eachof these instruction contexts to ensure that the command had the appropriateinterp retati on on the actual occasion of its use. In the next section, we show how the

semantic content of individual words is expressed in the terms of the set of modelsdefined in section 2.

Before moving to the analysis of particular English commands in the next section,a few general remarks must be made about grammars and parsers. No particulargrammar or parsing method is required for semantic trees. Howeve r, the concept ofa semantic tree does fit very well with the recent developments in augmented



C O N T E X T F I X IN G S E M A N T I C S 3 8 7

phrase-structure grammars that have led to t he theory o f lexical-functional gram-mars (LFG) and to the unification-based grammar formalisms such as D-PATR(Kaplan Bresnan, 1982; Shieber et al . , 1986). In our work on the arithmetic

instruction context, we have made use of the facilities of the LFG Workbench (aparser and set of debugging tools available on the Xerox 1108 Workstation) toimplement the context-free rules of our arithmetic instruction grammar as alexical-functional grammar (LF G User's Manual , 1985). A lexical-functional gram-mar assigns two levels of syntactic description to every sentence: a C-structure (forconstituent structure) which is a conventional phrase-structure tree, and anF-structure (for functional structure). In LFG theory, the purpose of the F-structureis to encode the meaningful grammatical relations of the sentence and to provideinformation about the predicate-argument structure of the sentence. Equationsattached to the category symbols on the right-hand side of the context-free rules ofthe grammar describe how to construct the F-structures. These equations specifyconditions of grammaticality for a sentence in addition to those provided by thephrase-structure rules. If all the conditions they specify are met, a well-formedF-structure is produced for the sentence. The C-structures delivered by theWorkbench for our instruction sentences were converted into semantic trees by themethod outlined above. Our only use of the F-structures was to help produce legalparses for the sentences by filtering C-structures. For a report of the lexical-functional grammar implemented for an extended corpus of arithmetic instructionsentences, see Crangle and Suppes (1985). Some details of the grammar for the

mobile robotic aid may be found in Michalowski, Crangle and Liang (1987).

4 Exam p le s o f con t ex t f ix ing

This section is organized around several examples, which are treated in considerabledetail. They concern commands to direct the agent's attention to various parts of anarithmetic exercise. That is, they are all l o o k - c o m m a n d s .We focus on this specificgroup of commands to acknowledge the many different natural ways there are ofgiving instruction on even one activity, and to examine the way the semanticapproach accommodates these different sentences of instruction. Grammatical rulesand lexical items are in troduced as needed by each example.

A brief explanation of notation is necessary. As before, we use POBJ for the setof perceptual objects in the arithmetic exercise facing the agent. We make theassumption that there is a one-to-one correspondence between the perceptual objectand the agent 's image of it and so use a, b, and c for the elements of POBJ and forimages of the elements of POBJ without differentiating between the two. Inaddition, we use the variables S and T for subsets of POBJ and R for binaryrelations on POBJ. These relations may be V ver t ica l ly be low )or H hor i zon ta l ly

lef t of ) or composite relations built up from V and H.

4 . 1 . L O O K T T H E T O P S P O T A N D L O O K T T H E T O P N U M B E R

The first two commands we examine in detail are L o o k a t t h e t o p s p o tand L o o k a tt h e t o p n u m b e r,with special emphasis on the adjective t o p and the way its preciseinterpretation is fixed both by its. immediate linguistic surround and by the



388 C CRANG LE AND P SUPPES

perceptual situation. The intuitive idea behind its semantic representation is bestunderstood if we talk about the top x where x is spo t in this sentence and would ber o w in the phrase the top row. When we say the top x, we are referring to that x

from a collection of entities of the same kind (that is, they are all x s) such that for agiven ordering relation R on these x s, the top x is the R-last entity. (The notion ofan R-last element of a set is defined as follows: a is the R-last element of a set S ifand only if for all b in S, b = a or (b, a ) E R.) The following procedure specificationcaptures these intuitive ideas for top.

( ) .RS){a : a ~ S (Ab)(if b e S then (b = a v (b, a) ~ R))}.

Values for S and R are provided by the context. In the simplest case, the noun thattop qualifies yields a value for S, and the perceptual situation provides a value for R.

To see how this lexical item is fixed by the context we must examine the word suse in a command. Consider the production rule and semantic function given belowin (1) for the phrase the top spot . We will discuss the interpretation of the top spotusing (1), and t hen show how a closer look at the con text demands a change in (1).The syntactic category of top is ADJdeg (for adjective of degree) while for s p o t andn u m b e r it is N (for noun). The semantic function in (1) shows that the denota tion ofthe noun provides a value for S in [top]. The other argument s value (an orderingrelation) is not given explicitly in the language of the command but must beprovided by the context. We use Rc for the relation R given by the context. Theappropriate relation is selected with the help of one of the registers, the one we have

labeled ORD . In keeping with the idea of an instructable robot, interaction with theuser will select the ordering relation the first time it is needed by the semantics. It isstored in ORD and subsequently used as the default unless verbal instructionexplicitly changes it. (Notice that our restricted mental model provides nomechanism for the perceptual situation itself to induce a change in the content ofORD.)

P roduc t ion ru le Seman t i c func t ion1 )

N P ~ the + ADJoeg + N, [NPI = [ADJd,g](Rc, [N]).

What is the appropriate ordering relation in the context of the arithmetic exercises?In the usual context of use for top, the agent has been, and is expected to continue,scanning down a column of figures. The relation V of being vertically below is thusexactly the one intended by these uses of top. Here then is a step-by-step analysis ofthe top spot .[the top spot]

= [ t o p ] R c , [ s p o t ] ) ,by rule (1),

= ( X R S ) { a : a ~ S (Vb)(if b E S then (b = a v (b, a ) ~ R))} (Rc , [spot]),

by the semantic representation of top,= {a: a E [spot] (Vb)(if b ~ [spot] then (b = a v (b, a) ERc))},

by function application,= {a: a ~ [spot] (Vb)(if b E [spot] then (b = a v b Va ) ) } ,

by the content of ORD.To see how this procedure specification for the top spot (or a similar one for the top




number) func t ions , we examine i t s u se i n a pa r t i cu l a r con tex t . Take the commandLook at the top number and c ons ide r the exerc i ses in F igs. 1 and 2 aga in , S uppo sethe agen t is f ocused som ew here i n the answ er row a t t he t ime the co m m and is g iven .

For the s ing le -co lumn addi t ion exerc i se in F ig . 1 , we would expec t tha t the agent i sbe ing aske d to lo ok a t the 7 . B ut for the two -colum n add i t ion in F ig . 2, is i t the 1 orthe 7? Th e re is o f cou r se n o w ay o f te l li ng f rom the com m and a lone . In f act , if t hep rocedure spec i f i ca t i on p roposed above fo rthe top spot were u sed in t hein te rpre ta t ion ofLook at the top spot fo r t he two-co lumn add i t i on exe rc i s e , noperceptua l ob jec t a t a l l would be iden t i f ied for there i s none ver t ica l ly above a l l theothers .

This i s where the contex t comes in to p lay once aga in . Ar i thmet ic ins t ruc t ion wi l lt yp ica ll y l ead the s tuden t dow n one co lumn a t a t ime . The con tex t t he re fo re na r rowsthe in t e rp re t a t i on o fspot to i nc lude on ly t hose pe rcep tua l ob j ec t s i n t he co lumn inwh ich the agen t i s p r e sen t ly focused . The commandLook at the top spot thus hasthe e ffec t o f Look at the top spot in this column, a c o m p l e t e l y u n a m b i g u o u scom m and in the a r i t hme t i c con tex t . H ow do we ensu re t ha t t he con tex t com es in top lay to l imi t the range of perceptua l ob jec t s? The answer l i es in recogniz ing tha t thein t ended in t e rp re t a t i on o fLook at the top spot is Look at the top spot in this column,tha t i s , in recogniz ing tha t the unqual i f ied phrasethe top spot i s semant ica l lyincomple t e i n the con tex t o f t he a r i t hme t i c i n s truc tion . W e the re fo re r ep l ace t heseman t ic func t ion in (1) by the fo l lowing .

[NP] = [AD Jd~g](Rc, [N] N Sc) ,whe re Sc , a s e t g iven by the co n tex t , i s i n t e r sec ted w i th t he d eno ta t i on o f the noun .W e th e r e f o r e h a v e t h e f o ll ow i n g a m e n d e d r u le ( l a ) .

Production rule Semantic function( l a )

NP--* the + ADJdcs + N , [NP] = [A D J d J (R c , [N] t l Sc ) .

Tak ing thi s app roach , we a l l ow an in te rp re t a t i on t o be g iven no t on ly fo rthe topspot bu t a l so fo r more complex ph ra se s such a sthe top spot to the left a n d the top

spot in the rightmost column. H e r e t h e p h r a s e sto the left and in the rightmost columnl imi t t he r ange o f pe rcep tua l ob j ec t s and p rov ide a va lue fo r Sc .To comple t e t he ana lys i s o fLook at the top spot, w e m u s t o f c o u r s e e x a m i n e t h e

seman t i c r ep re sen t a t i on o flook. I n t h i s c o m m a n d w e h a v e t h elook o f look at . . . .t he ' ob j ec t -o r i en t ed ' u se o flook. (The re a r e o the r u se s o flook. Ea r l i e r weme n t ioned the ' d i r ec t i on ' u se , a s inLook down until you see a bar. F o r o u rar i thmet ic ins t ruc t ion con tex t we the refo re iden t i fy two verb ca tegor ies : Vdd ford i r ec t ion and Vob j fo r ob j ec t -o r i en t ed . ) T he in tu i ti ve i dea beh ind the den o ta t i on o floOkobi i s tha t i t i s a p ro ce du re tha t p rod uce s a chang e of v isua l focus , f rompercep tua l o b j ec t a t o pe rcep tua l ob j e c t b , w he re b is t he t a rge t ob j ec t , t ha t is , t heob jec t t ha t i s l ooked a t . To cap tu re t he s eman t i c ro l e p l ayed by o the r words i n t hec o m m a n d , w e l e tlook deno te a func t ion f rom the u t t e r ance con tex t t o p rocedurespec i f ica t ions . The iex ica l en t ry forlook is therefore the fo l lowing:

Word Category Semantic representation

look, Vobj, (~.S){ (a , b ) : [SS], =a b ~ S SS



390 C CRANGL E AND P SUPPES

Th e ob jec t no un phrase y i e lds a va lue fo r the va r i ab le S. InLook a t the top spot ,t hed e n o t a t i o n o fthe top spotbecomes the va lue o f S ; i nLook a t t he t op number,t hed e n o t a t i o n o ft h e t o p n u m b e rbeco me s the va lue o f S. T he p roce dure spec if ica tion

for lookobj a l so inc ludes cond i t ions on the p rocedure ' s i npu t and ou tpu t , namely,tha t an image o f t he pe rcep tua l ob jec t a i s i n the SS reg i s t e r be fo re thel o o k - c o m m a n dis obe ye d ([SS]t = a ) and an im age o f t he t a rge t p e rcep tua l ob jec t bis p laced in the SS regis te r in obeying the command (SS~-- - , . I b ) . To parseL o o k a tthe top spot and to cons t ruc t a p rocedure spec i f i ca t ion fo r i t , we the re fo re r equ i rethe fo l lowing p rodu c t ion ru l e and sema n t i c func t ion .

Produc t ion ru le Seman t i c func t ion2 )

VP---~ Vobi a t N P, [ v P I = I Vo b , I I N P I ) .

T h e s e m a n t i c t r e e f o rL o o k a t t h e t o p s p o ti s g iven in F ig . 5 wi th the denota t iond i sp layed be low the l abel o f each non - t e rmina l node .A s i m i l a r c o m m a n d i s e x p r e s s e d b y t h e s e n t e n c eL o o k a t t he t o p n um b e r. T h e

w o r d n u m b e r i n the con tex t o f a r i t hme t i c in s t ruc t ion r e fe r s t o wha t we have ca l l ed' loca ted d ig i t s ' , tha t i s , the s ingle d ig i t s tha t appear a t a g iven t ime in a g ivena r i thme t i c exe rc i se . Because the agen t can focus on on ly one symbol a t a t ime (byour pe rcep tua l a s sumpt ions ) ,n u m b e r re fers only to the s ingle d ig i ts of the a r i thm et icexe rc i se , no t t o the numbers ( in the more gene ra l s ense o fn u m b e r t ha t occupyen t i r e rows . L o o k a t th e to p n u m b e ris pa r sed by the sam e p rodu c t ion ru l e s a sL o o kat the top spot . I f the agen t is focused on th e le f tm ost 6 of F ig . 6 ( tha t i s , SS conta insan image o f pe rcep tu a l ob jec t 624), t he r e su l t o f obey ing thi s com m and w i ll be tochang e the con ten t o f SS f rom 624 to 422 , t he n um ber in the g iven co lumn tha t has no

V P{ ( a , b ) : { S S ] = a b E [ t h e t o p s p o t I S S , ~ , b l

(kS ){ (a , b) : [SSlt= a b( S S S t+ b} N P

{a:a( [ ~ vbRC~))}N

the A D ,~eg {a: a~ POBJ}

( k R S} {a: a (S O'bXifb ( S the n (b = a v bRaD}

l o o t o p s p o t

L o o k a t t h e t o p s p o t

FIG. 5. Sem antic ree for sample comm and,



com-Ex'r-~x1NG SEMAmaCS 391

0 0 7

~o o 5

~o

o o o

FiG . 6 . A n exe rc i s e sh o w i ng a l l p e r c e p t u a l o b j ec t s .

other number vertically above it. L o o k a t t h e t o p s p o twould of course change theconten t of SS from 624 to o 21.

4.2. L O O K T T H E N E X T N U M B E R A N D L O O K T T H E N E X T S P O T

The word t o p is one of many words whose use presupposes some ordering relationon objects in the perceptual environment or on mental objects. Other words likethis are the ordinals f i r s t , s e c o n d , t h i r dand so on, and the word n e x t , which we willdiscuss in some detail in this subsection. We examine the two commands L o o k a t th en e x t n u m b e r and L o o k a t t h e n e x t sp o t.Once again, our emphasis is on the way theprecise interpretation of the word gets fixed by the perceptual situation and theimmediate linguistic surround.

The intuitive idea behind the semantic representation of n e x t can best beunderstood if we talk about t h e n e x t x where x may be n u m b e r , s p o t , or any othercommon noun. When we say t h e n e x t x , we are referring to the first x by some

ordering relation R (given by the context) relative to some present reference entity.So if R is the relation V of our perceptual situation, and if the present referenceentity is the perceptual object 7at of the exercise in Fig. 6, t he n e x t n u m b e rwill referto the perceptual object 8~2. If the present reference entity were 422, then t h e n e x tn u m b e r would refer to the perceptual object 624 while t h e n e x t s p o t would refer tothe perceptual object ~

The following procedure specification captures these intuitive ideas for nex t .

( X R S T ) { ( a , b ) :a 9 e T & T c S & ( b , a ) 9

& (Vc)(if c e T & ( c , a ) 9 R then (b = c v (c, b) 9 R))}.

The value of T is given by the denot ation of x of t h e n e x t x . Set-theoretical ly, it is asubset of S, where S is the set that is ordered by the relation R. We have to includethe superset S in the lexical entry and have the ordering relation on it rather than onT for the following reason. It makes perfect sense to say t h e n e x t x even when thepresent reference entity is not itself an x. For instance, the agent may be focused onthe perceptual object ~ which thus plays the role of the reference entity for theinterpretation of t h e n e x t n u m b e r . But o 23 and 624 must stand in the relat ion R for624 to be the first number in that relation. So R must be a relation on some superset

of the set of numbers. In the arithmetic context, that superset is the whole

Documents

International Journal of Man-Machine Studies Volume 27 Issue 4 1987 [Doi 10.1016%2Fs0020-7373%2887%2980004-4] Colleen Crangle; Patrick Suppes -- Context-fixing Semantics for Instructable