Computing with ShapesPaolo BottoniDepartment of Computer ScienceUniversity of Rome \La Sapienza"Via Salaria 113, 00198 Roma, ItalyGiancarlo MauriDepartment of Computer ScienceUniversity of MilanoVia Comelico 39, 20135 Milano, ItalyPiero MussioDepartment of Electronics for AutomationUniversity of BresciaVia Branze 38, 25123 Brescia, ItalyGheorghe P�aunInstitute of Mathematics of the Romanian AcademyPO Box 1 { 764, 70700 Bucure�sti, RomaniaTurku Centre for Computer ScienceTUCS Technical Report No 136October 1997ISBN 952-12-0088-XISSN 1239-1891

AbstractWe introduce a computability model { called shape grammar { based onadjoining polyominoes. A computation in a shape grammar is correct when:(1) it starts by using a speci�ed polyomino; (2) it ends when a rectangle isobtained (without holes); (3) at any step the current picture is connected;and (4) a sequencing mapping is given, so that at every step (except the �rstone) we use a polyomino depending on the previously used polyomino, asspeci�ed by this mapping (such a condition is essential for interactive visuallanguages, as formalized in [2], [3]). If strings of symbols are associated withthe polyominoes, then a string language can be associated to a computation.Surprisingly enough, in these circumstances we can characterize the re-cursively enumerable languages (hence the power of Turing machines). If wepreserve only conditions (1), (2), and (3) above, then we cannot generate alllinear languages but we can generate all regular languages and strictly more:also some one-letter non-regular languages can be obtained. In particular,we can obtain as correct computations squares only, which is often a di�culttask in picture languages (see, e.g., [32]).Keywords: visual languages, Chomsky hierarchy, equality sets of mor-phismsTUCS Research GroupMathematical Structures of Computer Science

1 IntroductionInteractive and end user computing are the present challenge of computerscience [5], [33], with the computer increasingly taking the role of a com-munication tool. Human-computer interaction (HCI) via a screen retains aspecial role in most interactive systems.A recently proposed approach to formal de�nition of screen-based HCIabstracts the arrangement of shapes on the screen as an image i, and themeaning associated with it in the interaction (which can also be the programcontrolling the interaction), as a string d, called the image description [3]. Arelation intmakes the correspondence between structures in i and symbols ind explicit, while a relation mat makes it explicit the inverse correspondencefrom d to i. Hence, a visual sentence is a quadruple vs = (i; d; int;mat). Ina human-computer interaction, visual sentences are generated as the resultof human actions and computer reactions. An Interactive Visual Language(IVL) is the set of visual sentences which may appear during an interaction.This approach seems general enough to encompass other proposed de�ni-tions of visual languages, derived by focusing on the parsing techniques forone-dimensional and graph languages (e.g. [13], [1]).The ability to relate structures in the image to symbols in the descrip-tions, and to understand the relations between the characteristics of visuallanguages and those of one-dimensional languages, are of capital impor-tance to properly extend to it IVLs the use of those mathematical andprogramming tools developed to classify, generate, parse and interpret one-dimensional sentences [27]. Problems arise in following this program becauseof the di�culties in characterising the algebraic structure of sets of images(it pictorial languages) and in relating these structures to those of one-dimensional languages. These problems have been explored in the �eld ofpattern recognition and image processing, since the seminal works of Kirsch[19] and Dacey [8], who studied the algebraic structure of pictorial languages,where strings are two-dimensional arrays with entities coloured by symbolsfrom an alphabet. Several characterisations of pictorial languages in purelytwo-dimensional terms have been de�ned so far (see e.g. [24], [25], [11]), buttheir practical usage appears to be limited.Other authors have avoided these di�culties by deriving linear descrip-tions (strings) of images and working on one-dimensional languages [31], [10].In this line, a set of pictorial structures are chosen as atomic and one or morerelations are chosen as signi�cant. Typically, atomic structures may be con-nected sets of pixels [31], or line segments [10], and a signi�cant relationis adjacency. In this second approach the strings of symbols (descriptions)are checked for membership in some formal language, using classical parsingmethods. However, a description focuses on a limited set of atomic elementtypes and relations, while images, especially in HCI, can be interpreted us-ing di�erent sets of atomic elements and relations [2]. The de�nition of IVL1

here adopted combines the advantages of the two approaches by allowingthe explicit management of both pictorial and description aspects and oftheir relations.In this paper we show that in an important case, when the image com-ponent of a visual sentence id treated as a puzzle [21], [23], a computabil-ity framework can be introduced that borrows elements from several 2Dlanguage manipulating formalisms. We use polygons consisting of four-connected pixels, i.e. unit squares which are orthogonally connected; weoperate on these atomic elements by adjoining them without overlapping;we preserve connectedness during the \computation"; and, as in interactivevisual languages used in man-computer communication, we impose a controlon the sequences of used polyominoes by means of a \next-move" mapping(this can be considered as a counterpart of the programmed string gram-mars, see [9]). A computation stops correctly when a rectangle is obtained.The combination of these conditions proves to be very powerful: every recur-sively enumerable language can be obtained by concatenating some stringsassociated with the used polyominoes.This result is proved by encoding in the visual formal system the char-acterization of recursively enumerable languages starting from equality setsof morphisms, [7], [29], [30]; we use here a variant of these results, recentlyproved in [18].As far as we know, our result is the \purest" characterization of recur-sively enumerable languages in a pictural framework: besides the controlby the next-polyomino mapping, no ingredient of a non-pictorial nature isused. In particular, we use no symbol or operation on symbols (as in the caseof array [24], puzzle [22], picture description [20] grammars), no matchingcolors as in the case of Wang tiles, etc.Our computability model can be seen as a variant of an interactive vi-sual language (IVL) in the sense of [2], [3]. Thus, the result above can beinterpreted as a proof of the computational completeness of IVL: the IVLis of the same power as Turing machines. This can have both a \positive"interpretation and a \negative" one: the IVL theory is very powerful, but,one can say, \too powerful". The undecidability of many questions can beinferred, because of this computational completeness. (A result like that in[4] is obtained as a consequence of our result here: the emptiness problemfor shape grammars is undecidable. It is worth noting that the proof in [4]makes use of a variant of a shape grammar, where one also uses stop poly-ominoes and the surface to be used is vertically bounded in advance. Theproof is based on a reduction to Post Correspondence Problem.)If the control on using the polyominoes is ignored, then a simple shapegrammar is obtained. Such apparently weak mechanisms can generate allregular languages; moreover, one-letter non-regular languages can be ob-tained (such languages are also non-context-free; in fact, they cannot be gen-erated by matrix/programmed grammars with arbitrary context-free rules2

but not using appearance checking, [16]). However, not every linear languagecan be generated by simple shape grammars.The study of shape grammars certainly deserves further e�orts.2 Formal language theory prerequisitesIn this section we recall only a few notions, notations, and results neededbelow. Further details can be found in [27] and in references therein.For an alphabet V , we denote by V � the free monoid generated by V ;� is the empty string, jxj is the length of x 2 V �. The language of allnon-empty strings over V , that is V ��f�g, is denoted by V +. A morphismh : V � �! U� such that h(a) 2 U [f�g for all a 2 V is called a weak coding;it is called a coding when h(a) 2 U for all a 2 V and a projection whenh(a) 2 fa; �g for all a 2 V .A Chomsky grammar is a construct G = (N; T; S; P ), where N; T aredisjoint alphabets, S 2 N , and P is a �nite subset of (N [T )�N(N [ T )��(N [ T )�. The elements of N are called nonterminal symbols, those of Tare called terminal symbols, S is the axiom, and the elements (u; v) 2 P ,written in the form u ! v; u; v 2 (N [ T )�, are called rewriting rules (forshort, rules). The language generated by G is denoted L(G):We denote by FIN;REG;LIN;CF;CS;RE the families of �nite, reg-ular, linear, context-free, context-sensitive, and recursively enumerable lan-guages, respectively.A �nite automaton is a construct M = (K; V; s0; F; �), where K; V aredisjoint alphabets (the set of states and the input alphabet), s0 2 K (theinitial state), F � K (�nal states), and � : K � V ! 2K (the transitionmapping). We denote by L(M) the language recognized by M . One knowsthat �nite automata (in the form above or even deterministic, that is with�(s; a) 2 K for each s 2 K; a 2 V ) characterize the regular languages.A result which will be esentially used here is the characterization ofrecursively enumerable languages starting from equality sets of morphisms.For two morphisms h1; h2 : V � �! U� we denoteEQ(h1; h2) = fw 2 V � j h1(w) = h2(w)g:Theorem 1. ([7], [29], [30]) For every language L 2 RE;L � V �, thereare two alphabets V1; V2 such that V � V1, two �-free morphisms h1; h2 :V �1 �! V �2 , a regular set R � V �1 , and a projection h3 : V �1 �! V �, suchthat L = h3(EQ(h1; h2) \R).Theorem 2. ([18]) For every language L 2 RE;L � V �, there are twoalphabets V1; V2 such that V � V2, two �-free morphisms h1; h2 : V �1 �! V �2 ,a regular set R � V �2 , and a projection h3 : V �2 �! V �, such that L =h3(h1(EQ(h1; h2)) \R). 3

Note the di�erence between the two results: in the second case we usethe image of the equality set of h1; h2 through one of the two morphismsthemselves. It is also worth noting that h1; h2 are �-free and that h3 is aprojection.Convention. Two languages are considered equal if they di�er at mostby the empty string: L1 = L2 if and only if L1 � f�g = L2 � f�g.3 Shape grammarsFirst, we specify some notions concerning the basic objects we use, thepolyominoes.We work on the in�nite two-dimensional grid of integer coordinates(without making use of coordinates: a picture is identi�ed by its shape,not by its position in the plane). The elementary object we use is the pixel,the unit square of the grid. By orthogonally adjoining pixels we obtain poly-ominoes. Therefore, a polyomino is a connected (in the orthogonal mode,horizontally and vertically) region of the plane. Figure 1 presents exam-ples of polyominoes (a, b) and of non-polyominoes (c and d). Note thatthe polyomino b contains a \hole" (the black pixel). We allow such holesof any size and shape as long as the �gure remains connected in the sensementioned above.Figure 1: Examples of polyominoes and non-polyominoesa b c dThe polyominoes can be adjoined in such a way that at least one unitsegment from the edge of each of them is identi�ed, but no pixel of the twopolyominoes is superposed. In general, for two given polyominoes many waysof adjoining them are possible. So, several polyominoes can be obtained byadjoining two given polyominoes p1; p2. We denote by adj(p1; p2) the setof all such polyominoes. We shall not give a more precise de�nition ofthis operation (for instance, by specifying coordinates of the terms of theoperation and of the resulting polyomino).We can now introduce the main notion of our investigation.A shape grammar is a system = (V; P; p0; next; lab);4

where V is an alphabet, P is a �nite set of polyominoes, p0 2 P; next :P �! 2P , and lab : P �! V �.The polyomino p0 is the initial one, next is the mapping controlling thesequence of using the polyominoes in P , and lab is the labeling mapping, aparticular case of int in the sense of [3].A computation in is a sequence� = p0p1p2 : : : pn; n � 1;of polyominoes in P such that the following conditions are ful�lled:1) there is a rectangle which can be covered by p0; p1; : : : ; pn, completely(i.e.without leaving holes) and without overlapping, and there is norectangle covered by a subsequence p0; p1; : : : ; pk, k � n.2) when building this rectangle one uses the polyominoes in the orderp0; p1; : : : ; pn by adjoining (thus, at every step the current picture is apolyomino, that is it is connected),3) pi 2 next(pi�1); 1 � i � n:The set of all rectangles obtained at the end of computations in isdenoted by pic( ). To each computation � as above we can also associatea string over V , namely lab(�) = lab(p0)lab(p1) : : : lab(pn). The language ofall such strings associated to computations in is denoted by L( ). We saythat this is the language generated by .We denote by SL the family of all languages of this form.A shape grammar = (V; P; p0; next; lab) such that next(p) = P for allp 2 P is said to be simple. Because the mapping next imposes no restrictionin such a case, we omit it and we write in the form (V; P; p0; lab). However,without the control of the mapping next, after completing a computation wecan always start another one (in particular, the same), which, by iteratingthis procedure, will possibly lead to a rectangle composed of several rectan-gles describing correct computations. Consequently, the generated languagewill be closed under Kleene +. In order to avoid this, in the case of simpleshape grammars we impose the additional condition that p0 is used onlyonce during a computation, in the �rst step.We denote by SSL the family of languages generated in this way bysimple shape grammars.If the labeling mapping lab takes values in V [ f�g, then we say that = (V; P; p0; next; lab) is reduced. We denote by RSL;RSSL the familiesof languages generated by reduced and by reduced simple shape grammars.Of course, it is possible that a sequence � = p0p1 : : :pn can lead toseveral rectangles and, conversely, the same rectangle can be decomposed5

in several sequences of polyominoes which constitute correct computations.This will not be important for the results in the subsequent sections, but fora closer study of languages generated by shape grammars and of their prop-erties (what about ambiguity ?), a more precise de�nition of the languagegenerated should be stated (for instance, specifying the position where thepolyominoes are placed and adjoined).We illustrate the de�nition (and the discussion above) by two simpleexamples, proving the assertion: each �nite language can be generated bya reduced simple shape grammar.Figure 2: Polyominoes used in Example 16?

6?kmkij iiiij � 11a b c dExample 1. Consider an alphabet V = fa1; : : : ; ang and a �nite lan-guage L � V �; L = fw1; : : : ; wmg. For each wi = ai;1ai;2 : : : ai;ki , ki � 1(remember that we ignore the empty string), ai;j 2 V; 1 � j � ki; 1 � i � m,we consider the polyominoes pi;j for j = 2; 3; : : : ; ki� 1, as shown in Figure2c, where k = maxfki j 1 � i � mg; we also consider the polyominoes p0(Figure 2a), pi;1 (Figure 2b), and pi;ki (Figure 2d). The pixels marked withi are placed in the ith row of pixels above the lower dotted horizontal line,the pixel marked with j � 1 is placed in the (j � 1)th row of pixels abovethe middle dotted horizontal line, and similarly for the pixels marked withj and ki.One sees how the \hanging" pixels correspond to missing pixels in oppo-site edges and in edges of other polyominoes, making it possible the correctadjoining of polyominoes in a unique way. Moreover, the lower \hanging"pixels paired with holes identify the string wi and the upper \hanging" pix-els identify the position in the string wi (hence the symbol in that position).With P consisting of all these polyominoes, we consider the shape grammar 1 = (V; P; p0; lab), with lab(p0) = �; lab(pi;j) = ai;j ; 1 � i � m; 1 � j � ki.6

The equality L( 1) = L is now obvious: after using p0, the only possibleadjoining such that a connected picture without holes is obtained at the endis to use a polyomino pi;1; this forces the use of polyominoes pi;2; : : : ; pi;ki�1,in this order, at the subsequent steps and then of pi;ki ; in that moment weobtain a rectangle and no continuation is possible, because p0 cannot beused again. (Note the fact that this condition is essential, otherwise we getall strings in L+.)Example 2. When using simple but not necessarily reduced shapegrammars, we can generate the �nite language L in an easier way: considerp00 as in Figure 3a and a polyomino pi associated with wi for each i =1; 2; : : :m, as in Figure 3b.Figure 3: Polyominoes used in Example 2� -ia bFor the shape grammar 2 = (V; fpi j 1 � i � mg [ fp00g; p00; lab); withlab(pi) = wi; 1 � i � m, and lab(p00) = �, we obviously get L( 2) = L.Examples of shape grammars will also appear in the proofs in the nextsections.4 Preliminary resultsDirectly from the de�nitions we obtain the relations below:Lemma 1. RSSL � (SSL\ RSL); (SSL[ RSL) � SL:From Example 1 we also know that FIN � RSSL. This relation willbe signi�cantly strenghtened in Section 6.The inclusions in Lemma 1 are not necessarily proper. More precisely,the restriction in reduced shape grammars does not decrease the generativepower of general shape grammars.Lemma 2. SL � RSL.Proof. Consider a shape grammar = (V; P; p0; next; lab) such thatlab : P �! V � and there are polyominoes p 2 P such that jlab(p)j � 2.Assume P = fp0; p1; : : : ; pmg and lab(pi) = wi; 0 � i � m, with jlab(pi)j= ki. We denote k =maxfki j 0 � i � mg. Assume also V = fa1; a2; : : : ; angand wi = ai;1ai;2 . . .ai;ki .Every polyomino contains at least one pixel. For the sake of reasoning,let us suppose that we can change the length unit, decreasing it by a factor7

of 3. After this change, each polyomino will contain at least one square3� 3, that is a pixel which is completely surrounded by other pixels in thepolyomino. Let us now decrease again the unit, by a factor of t, wheret = max(m+ k + 1; 3k):Then, every polyomino will contain a square of size t � t completely sur-rounded by used pixels; Figure 4 illustrates the idea.Figure 4: Decreasing the measure unit 6?t=) =)In this way, the shape of polyominoes in P is not changed. (The readeris easily convinced that in the case of �xed length unit, so that it cannotbe changed, then we can proceed in the symmetric way, considering newpolyominoes, of sizes which are t times larger than the sizes of initial poly-ominoes. The argument below runs in the same way for such a case.) Forevery polyomino pi 2 P; 1 � i � m; we consider the following polyominoes:{ the polyomino p00, obtained by removing from pi an internal rectangleof size (k+m)� 3ki, plus some adjacent pixels as suggested by Figure5 and as they will be precisely speci�ed below;{ the polyominoes of the form in Figure 6, for each j = 1; 2; : : : ; ki; letus denote them by pi;j .The idea is to �ll-in the hole in p0i with the new polyominoes, one for eachsymbol in wi, in the same way as we have done in Example 1. Thus, the leftedge of the hole plays the role of the domino b in Figure 2, identifying thestring wi and also imposing the adjoining of a polyomino associated withthe �rst symbol of wi; polyominoes of the form in Figure 6 play the role ofpolyominoes of type c in Figure 2, and the right edge of the hole plays therole of the polyominoes of the form d in Figure 2. The new polyominoesalso have an \hanging" pixel in the lower edge, meant to �ll-in the holes inthe lower edge of the hole of p0i and to prevent the use of these polyominoesoutside such holes: no polyomino or sequence of polyominoes �ts the shapeof the lower edge of the new polyominoes, except the lower edge of holes inpolyominoes p0i. 8

We construct the reduced shape grammar 0 = (V; P 0; p00; next0; lab0),where P 0 contains all polyominoes described above, p00 corresponds to p0,and the mappings next, lab are de�ned as follows:next0(p0i) = fpi;1g; 0 � i � m;next0(pi;j) = fpi;j+1g; 0 � i � m; 1 � j � ki � 1;next0(pi;ki) = fp0j j pj 2 next(pi)g; 0 � i; j � m;lab0(p0i) = �; 0 � i � m;lab0(pi;j) = ai;j ; 0 � i � m; 1 � j � ki:Figure 5: The shape of the hole in initial polyominoes

6?6?km 1i i2 iki

Figure 6: New polyominoes in the proof of Lemma 26??6km i ijj-1

From the construction of 0 it is clear that 0 can compute the samerectangles as , but of size multiplied by 3t and using new polyominoes;9

if �0 is a computation in 0 corresponding to a computation � in , thenlab(�) = lab0(�0). Therefore, L( ) = L( 0). Because 0 is reduced, weobtain the inclusion SL � RSL.Corollary 1. SL = RSL.Remark 1. The above proof does not work in the case of simple shapegrammars: the \small" polyominoes pi;j can be used at any time after usingp0i and in any order (the picture is connected and no restriction on the orderof using the polyominoes is imposed), so we can get a computation for anypermutation of wi (and even with permuted occurrences of strings wi).The problem whether or not the inclusion RSSL � SSL is proper re-mains open.5 A characterization of recursively enumerable lan-guagesWe give now the main result of this paper.Theorem 3. RE = SL = RSL:Proof. We invoke the Turing-Church thesis for the inclusion RSL �RE; actually, we have no reasonably simple proof of this inclusion (themain di�culty lies, of course, in the passing from two-dimensional �gures tostrings of symbols; a possibility is to use two-dimensional Turing machines,in the sense of [17], but we leave the details to the reader).In order to prove the inclusion RE � SL we will use the characterizationof recursively enumerable languages in Theorem 2.Consider a language L 2 RE;L � V �, the projection h3 : V2 �! V [f�g,the morphisms h1; h2 : V �1 �! V �2 , and the regular language R � V �2 ,such that L = h3(h1(EQ(h1; h2)) \ R). Assume that V1 = fa1; : : : ; angand V2 = fb1; : : : ; bmg, and consider a deterministic �nite automaton M =(K; V2; s0; F; �) such that L(M) = R; assume that K = fs0; s1; : : : ; srg.We construct �rst the primitive polyominoes p(bi) in Figure 7, 1 � i � m,as well as the primitive polyominoes p(sj ; bi; sk) in Figure 8, 0 � j; k � r; 1 �i � m. The pixel marked with i in Figure 7 is placed in the ith position fromthe left end of the polyomino; the same holds for the pixel marked with i inthe lowest row of pixels of the polyomino in Figure 8. However, the pixel jin Figure 8 is placed in the (j + 2)th row, counting from the �rst bottomrow of pixels: one row is reserved to holes as that marked with i, the nextrow corresponds to s0 2 K, hence it is associated with the digit 0, and soon; this also holds for the pixel marked with k.We now construct the shape grammar = (V; P; p0; next; lab)10

as follows. The polyomino p0 is as shown in Figure 9.For each ai; 1 � i � n; in V1 such that h1(ai) = bj1bj2 : : : bjk we con-sider a polyomino p(ai) obtained by adjoining the primitive polyominoesp(bj1); p(bj2); : : : ; p(bjk), such that they are horizontally placed on the sameline. Figure 7: Primitive polyominoes p(bi)-� miFigure 8: Primitive polyominoes p(sj ; bi; sk)

� - 6?r + 1kmj iMoreover, for each ai; 1 � i � n, in V1 such that h2(ai) = bl1bl2 : : : blt, weconsider all polyominoes p(sj ; ai; sk); 0 � j; k � r, obtained by consideringthe sequence sq1 ; : : : ; sqt�1 of states of M such that sq1 = �(sj ; bl1); squ =�(squ�1 ; blu); for 2 � u � t� 1, sk = �(sqt�1 ; blt), and adjoining the primitivepolyominoes p(sj ; bl1; sq1); p(sq1; bl2; sq2); : : : ; p(sqt�1; blt; sk):Again, we place the polyominoes straight in a line, with the upper edge onthe same line.Therefore, we consider all possibilities to parse the string h2(ai) in theautomaton M starting in a state sj and arriving in a state sk, for all sj ; sksuch that sk = ��(sj ; h2(ai)), where �� : K � V �2 �! K indicates thetransitive closure of � on strings of symbols. All polyominoes p(sj ; ai; sk)associated to ai di�er only in their upper part (the upper r + 1 rows ofpixels) and have identical lower parts (the lowest row of pixels); this lowestrow precisely identi�es the string bl1bl2 : : : blt = h2(ai), whereas the upperpart describes a possible parsing of this string in M .11

The set P contains all polyominoes p0; p(ai); p(sj; ai; sk) constructed inthis way, as well as all polyominoes p(sk); sk 2 F , as shown in Figure 10.The mapping next is de�ned as follows:next(p0) = fp(ai) j 1 � i � ng;next(p(ai)) = fp(sj ; ai; sk) j sj ; sk 2 K; sk = ��(sj ; h2(ai))g; 1 � i � n;next(p(sj ; ai; sk)) = fp(au) j 1 � u � ng [ fp(sq) j sq 2 Fg;1 � i � n; 0 � j; k � r;next(p(sk)) = ;; sk 2 F;whereas the labeling mapping lab is de�ned bylab(p0) = lab(p(sf)) = lab(p(sj; ai; sk)) = �;for sf 2 F; 1 � i � n; 0 � j; k � r;lab(p(ai)) = h3(h1(ai)); 1 � i � n:Figure 9: Polyomino p0 Figure 10: Polyominoes p(sk)

6 0 6?r + 1k?r + 1For this shape grammar we have the equality L = L( ).This can be easily seen from the shape of polyominoes in P and thede�nition of mappings next and lab. We start to build a rectangle by usingthe polyomino p0. Its left vertical edge will be the left vertical edge of therectangle. The polyomino p0 introduces the state s0 of the automaton Mrecognizing the language R. We continue by alternately adjoining polyomi-noes p(ai) and p(sj ; ai; sk). Due to the shape of edges, there is a uniqueplace where these pairs of polyominoes can be adjoined in such a way asto �nally obtain a connected picture without holes and placed inside therectangle whose left vertical edge is already �xed. Thus, we continue fromleft to right, with polyominoes p(ai) along the lower horizontal edge of therectangle and the polyominoes p(sj ; ai; sk) placed along the upper horizontal12

edge. A rectangle can be completed only by using a polyomino p(sk); sk 2 F ,which will draw the right vertical edge.Consider a computation� = p0p(ai1)p(sj1 ; ai1; sk1)p(ai2)p(sj2 ; ai2 ; sk2) : : :p(ait)p(sjt; ait ; sf)p(sf):Because when using p(sf) the computation should be �nished (by producinga complete rectangle), we must havejh1(ai1ai2 : : : ait)j = jh2(ai1ai2 : : : ait)j(modulo the \hanging" pixels, all primitives have the same width, m =card(V2)).Because of the \hanging" pixels in the upper edge of polyominoes p(ai)and the holes in the lower edge of polyominoes p(sj ; ai; sk), in fact we musthave h1(ai1ai2 : : :ait) = h2(ai1ai2 : : : ait):This means that ai1ai2 : : : ait 2 EQ(h1; h2), that is h1(ai1ai2 : : :ait) 2h1(EQ(h1; h2)).Moreover, the \hanging" pixels and the holes in the upper r + 1 rowsof pixels in polyominoes p(sj ; ai; sk) check the correct parsing of the stringh1(ai1ai2 : : : ait) by the automatonM ; the condition to �nish in a �nal stateis ensured by the polyomino p(sf). That is, h1(ai1ai2 : : :ait) 2 R. In sum,h1(ai1ai2 : : : ait) 2 h1(EQ(h1; h2))\R.The labeling mapping lab \ignores" all polyominoes di�erent from p(ai1);. . . , p(ait) and assigns to these polyominoes the strings h3(h1(ai1)); . . . ,h3(h1(ait)). Consequently, lab(�) = h3(h1(ai1ai2 : : : ait)): This means thatthe string identi�ed by the computation � is in L, that is L( ) � L.From the previous discussion it is easy to see that for each string in L wecan �nd a computation in producing it, hence also the converse inclusionis true.6 The power of simple shape grammarsThe key feature of shape grammars allowing us to \paint" an equality set oftwo morphisms by using polyominoes adjoining in the proof above was themapping next. Using it we have ensured that when building a rectangle inwhich two sequences of polyominoes match each other, these two sequencescorrespond to the same string in the domain of h1; h2. (The matching cor-responds to the equality h1(w1) = h2(w2) and it is checked by the shape ofpolyominoes, the equality w1 = w2 is ensured by the mapping next.) Thus, itis expected that simple shape grammars are much weaker that unrestrictedshape grammars. We shall show that this is the case indeed, after proving13

that reduced simple shape grammars are not \too weak": they can generateall regular languages and something else (squares, which is quite signi�cantfor the power of our computing framework).Theorem 4. REG � RSSL:Proof. Consider a deterministic �nite automaton M = (K; V; s0; F; �)with K = fs0; s1; : : : ; smg and V = fa1; a2; : : : ; ang. We construct theshape grammar = (V; P; p0; lab), containing the following polyominoes:{ p0 is indicated in Figure 11a (so, from below, the rows are assigned to0; 1; : : : ; m);{ for every triple (sj ; ai; sk) 2 K � V � K such that sk = �(sj ; ai), weconsider a polyomino p(sj ; ai; sk) as that in Figure 11b;{ for each sk 2 F we consider a polyomino p(sk) as that in Figure 11c.Figure 11: Polyominoes used in the proof of Theorem 4

-� 6?m+ 1i0 j k ka b cThe labeling mapping is de�ned bylab(p0) = lab(p(sk)) = �; sk 2 F;lab(p(sj; ai; sk)) = ai; 0 � j; k � m; 1 � i � n:The equality L( ) = L(M) is obvious: the shape of polyominoes makespossible the construction of rectangles which describe computations of M .(This is exactly the idea used in the upper part of rectangles constructed inthe proof of Theorem 3.)Note that is both simple and reduced.The inclusionin Theorem 4 is proper. In fact, rather complex languageswith respect to the Chomsky hierarchy can be generated by reduced simpleshape grammars, namely one-letter non-regular languages; such languagesare also non-context-free. 14

Theorem 5. RSSL� CF 6= ;:Proof. Let us consider the reduced simple shape grammar = (fag; fp0; p1; p2; p3; p4; p5g; p0; lab);with the polyominoes indicated in Figure 12 and with the mapping lab de-�ned by lab(pi) = a; 0 � i � 5:

Figure 12: Polyominoes used in the proof of Theorem 5p3

p1 p2p0 p4 p5The idea of this construction is the following. The polyominoes p2; p3have \long" segments on their edges, the polyominoes p4; p5 have \short"segments on their edges. Moreover, the polyomino p2 has pixels missing ina corner, thus p5 cannot be adjoined in that place. The only way to obtainthe corners of the rectangle is to use p0 in the upper right corner, p3 inthe lower right corner, p1 in the lower left corner, and p5 in the upper leftcorner. This means producing \short" segments in the upper left corner ofthe rectangle and \long" segments in the lower right corner. Because an edgecomposed of \small" segments cannot be adjoined to polyominoes p2; p3, we15

need an intermediate polyomino; this is p1, which has \small" segments onthe north-west edge and \long" segments towards south-east. Thus, we areforced to use copies of the polyomino p1 on the SW { NE diagonal of therectangle, which in this way is forced to become a square. Moreover, afterplacing p0, only p1 can �t it from below, and this forces the use of p3 tothe right of p1. Under the obtained �gure we have to use only polyominoesp2; p3: this is the only way to get a straight line in the right hand of therectangle (remember that, because is simple, we are not allowed to use p0again).If, after completing a rectangle we intend to continue { without usingp0 { then no new rectangle can be obtained: we need to use p1 in order to\intermediate" between other polyominoes; to each p1 we have to adjoin onep2 or one p3; both p2 and p3 are of an even vertical size, whereas p1 is of anodd vertical size; sliding various copies of p1 with respect to each other inorder to adjust the parity will make the covering of the rectangle impossible(either the missing pixel in the angle of p1 will be blocked, or near it anothersmall segment will appear, and no possibility exists to match it).In short, we can only produce squares. They look as shown in Figure 13(where two copies of p1 are used). Clearly, squares in which any number ofcopies of p1 are used can be constructed. When n copies of p1 are used, thesize of the square is (8n+ 4)� (8n+ 4); n � 1:Let us see which is the string associated to a square of this size. We use:{ one copy of p0;{ n copies of p1;{ n copies of p3;{ n�1Xi=1 i = n(n� 1)2 copies of p2;{ 2(n� 1) + 2 = 2n copies of p5;{ 2(n�1)Xi=1 i+ 2(n� 1) = 2n2 � n � 1 copies of p4.Each polyomino is labeled with a, hence we get a string am withm = 1 + n + n+ n(n� 1)2 + 2n+ 2n2 � n � 1= 5n(n+ 1)2 : 16

(Notice that this is indeed an integer number.) In conclusion,L( ) = fa 5n(n+1)2 j n � 1g;which is not a regular language. As each one-letter context-free language isregular, L( ) =2 CF either.

Figure 13: An example of a square in the proof of Theorem 5Let us denote by PR� the family of languages generated by programmedgrammars with arbitrary context-free rules but without appearance check-ing, [26], [9].Corollary 2. RSSL� PR� 6= ;:Proof. In [16] it is proved that every one-letter language in the familyPR� is regular.Remark 2. If the reader wants to check whether the number of copiesof polyominoes speci�ed above are indeed those appearing in a square ofsize (8n+ 4)� (8n+ 4), one can compute the surface of these polyominoes.Because the surfaces of p0; p1; p2; p3; p4; p5 are equal to 32, 42, 64, 50, 16, 10,respectively, the surface of used polyominoes isS = 32+ 42 � n+ 50 � n+ 64 � n(n� 1)2 + 10 � 2 � n+ 16 � (2n2 � n� 1):17

By a simple calculation we obtainS = 64n2 + 64n+ 16;that is, the surface of the square.Theorem 6. LIN � SSL 6= ;:Proof. Let us consider the linear (non-regular) languageL = fanbn j n � 1gand assume that L = L( ) for some simple shape grammar = (fa; bg; P; p0;lab). Consider a string anbn with arbitrarily large n; let � be a computationin such that lab(�) = anbn. It follows that a rectangle D, of arbitrarilylarge size on at least one of the two directions is constructed when following�. Let us denote by P 0 the set of polyominoes p 2 P such that lab(p) 6= �.Arbitrarily many elements of P 0 are used in the construction of D. If strictlymore than card(P 0) of them are reaching two opposite edges of D, as inFigure 14, then one of them is repeated, hence a subpicture D0 of D can beiterated and still we produce rectangles.Figure 14: The possibility to iterate a subpictureof D, in the proof of Theorem 6@@ ��D0�� @@D HHHHHHHHHHHY BBBBBBMrepeatedpolyominoesThis means that a non-empty substring of anbn can be iterated, whichleads to strings not in L, a contradiction.Consequently, only a bounded number of polyominoes in P 0 can reachtwo opposite edges of the rectangle (maybe none). This means that thereare arbitrarily many polyominoes in P 0 which touch at most one edge fromeach pair of opposite edges of D. Let us choose the �rst polyomino inthe computation � associated with D, di�erent from p0 and having thisproperty; let us denote it by p1. Because n is arbitrarily large, we must havelab(p1) 2 a+. Removing p1 from D we obtain a connected picture. Becausep1 does not touch two opposite edges of D, we can distinguish two cases: (1)p1 has at least two neighbors which are adjacent to each other; (2) p1 has18

only one neighbour. In case (1) we can change the order of adjoining thepolyominoes used in the computation � in the following way: we start as in� and we proceed as in � until adjoining the polyomino preceding p1 in �;we omit p1 and continue with a neighbor of p1 in D in such a way that aconnected picture is obtained; we complete the construction of the rectangleD minus p1 in such a way that at each stage the �gure is connected (thisis possible, because D is constructed during the computation � observingthis restriction and p1 has two neighbors which are adjacent to each other;because is simple, no restriction on the sequence of used polyominoes isimposed); at the end, we also add p1. In this way, we obtain the samerectangle D by a computation which ends by adjoining the polyomino p1.Therefore, the string associated with this new computation ends with asu�x ai; i � 1. Such a string is not in L, a contradiction. In case (2),the polyomino p1 is "surrounded" by another polyomino, let us denote itby p2, which separates it from the rest of the rectangle. This means thatp1 is introduced after p2, and that after introducing p1 we use a polyominoadjacent to p2. Therefore, after skipping p1 we can continue the processas in the computation of D. Finally, we also add p1, and we get the samecontradiction as above. Consequently, L =2 SSL.Corollary 3. Each of the families RSSL, SSL are incomparable witheach of LIN and CF.Prof. Combine Theorems 5 and 6.In fact, we can state a more general assertion:Corollary 4. Each family RSSL, SSL is incomparable with any familyF such that LIN � F � PR�.Proof. Combine Corollary 2 with Theorem 6.Corollary 5. The inclusion SSL � SL is proper.Synthesizing in a pictorial way the results in Sections 4, 5, 6 (as wellas the known inclusions in the Chomsky hierarchy), we get the diagram inFigure 15; an arrow from a family F1 to a family F2 indicates the inclusionF1 � F2. The inclusion RSSL � SSL is not known to be proper, all otherinclusions in the diagram are proper.7 CommentsWe have mentioned in the Introduction the relation between the computabil-ity model considered here and the interactive visual languages (IVL) intro-duced and discussed in [2], [3]. In view of Theorems 3, 5, this relationdeserves to be emphasized. The shape grammars can be seen as a particulartype of an IVL, which means that the premises of these theorems are ful-�lled by the \complete" IVL. Thus, the power of IVL can be considered as19

covering the power of shape grammars. Stated directly, the IVL frameworkis computationally complete, hence most decidability problems about it arenot algorithmically solvable.Figure 15: The hierarchy of shape languagesin comparison with the Chomsky hierarchyREGZZZZZ} �����3LINRSSL6SSL���� @@@I CSCFSL = RSL = RE 66

As we have remarked, this power stands in the possibility of doublechecking of the correctness of pictures: at the syntactic level (by matchingshapes) and at the semantic level: in IVL, the icons used in the human-computer communication must have the same meaning for the two partners,in order to ensure a sucessful communication; moreover, the communicationconsists of a sequence of human orders and computer replies, both of themexpressed in the form of icons (thus, a relation of the type of the nextmapping is necessary). In the shape grammars, this \semantic" restrictionis ensured by the next mapping, which allows continuing a computationonly with a prescribed set of polyominoes, for instance, those with the same\meaning" as the icons in a human-computer communication via an IVL.The fact that reduced simple shape grammars can generate non-context-free languages (even not in PR�) can be interpreted as a further indicationabout the power of \computing by shapes". Even without the control pro-vided by the mapping next, we can generate complex languages.It is worth mentioning that our method of generating squares remindsthe problem open for a while whether or not regular (or context-free) isotonicarray grammars can generate solid rectangles and squares. Although we donot use here rewriting rules, hence nonterminals controlling in a symbolicway the continuation of a computation, we have obtained squares in aneasier way than that in [32].It is also worth emphasizing the similarity of our characterization of re-cursively enumerable languages by means of shape grammars (Theorem 5)with the characterization of recursively enumerable languages by the so-20

called coherent regular simple sticker systems in [18]. In [18] one computesby building DNA molecules by annealing single stranded sequences of nu-cleotides. A computation stops when a molecule is produced and it is co-herent when the upper level blocks have the same labels as the lower levelblocks used in the computation. This is exactly the semantic condition inIVL { it can be ensured in shape grammars by the mapping next: the pairsof icons have the same semantic mark, in the sticker systems the pairs ofblocks have the same label. The syntactic condition of matching adjacentedges is provided in the DNA case by the Watson-Crick complementarity.Concerning the stop condition of a computation, one can imagine severalvariants di�erent from that considered here. A powerful one is to specifyalso the last polyomino to be used, not only the �rst one. As in [4], we canimpose a limit on the height of the obtained picture, and then the adjoiningof polyominoes is signi�cantly restricted. We can also impose domino-likeconditions, controlling at the same time the adjoining and the terminationof the process. For instance, we can consider colors associated with unitarysegments on polyomino edges, as in Wang tiles, or only a head segmentand a tail segment, as in [31]. Furthermore, inspired from DNA area, wecan designate some segments as sticky. In all latter cases, we can allow anadjoining only if we match colors, we match a head with a tail, or we matchtwo sticky segments, respectively, and we may allow �nishing the processonly when no free colour, or free sticky segment is available. Another stopcondition can be of the type of adult languages in Lindenmayer systemsarea: stop when no further step is possible (we can call such computationsstationary of blocked). Most of these variants are stronger than that ofstopping when a rectangle is produced. Anyway, computing in a frameworkwhere such features are present { possibly added to a shape grammar asconsidered here { will lead again to results like Theorems 3, 4, 5. Computingby shapes is powerful in many variants. And, it is important to note, suchfeatures (conditions for correct adjoining) appear in many papers devoted tovisual languages (e.g [21], [23]). Adding to the models considered in thesepapers a control of the type of the mapping next, in many cases we willget computational completeness. A systematic checking of this assertiondeserves to be carried out.It is also important to repeat the fact that the control by a next mappingis very natural: it appears, directly or in disguise, in many classes of regu-lated rewriting grammars, [9], starting with the programmed grammars of[26], and it also appears, in a somewhat weaker variant, in IVL, through thepairing of human commands and the computer answers. However, in theproof of Theorem 3, such a control of the adjoining polyominoes is su�cient(see again the de�nition of the mapping next in the proof of Theorem 3).Computing in terms of shape grammars also suggest one new measureof computational complexity: the surface of a computation, compared tothe length of the produced string. This is related to the duration of the21

computation (the number of used polyominoes), but it can give an additionalinformation about the di�culty of computing a language. (This measure alsoreminds of the weight of a computation, proposed in [15] as a measure ofthe complexity of computing with DNA.)For instance, from the construction in the proof of Theorem 4, we �ndthat a string w in a regular language L(M), recognized by a deterministicautomaton M = (K; V; s0; F; �), is computed using a surface equal toS(jwj) = 2 � (card(K) + 1) + jwj � card(V ) � (card(K) + 1) + (card(K) + 1):Dividing by jwj, for w of large length we �nd that the computation is donein a surface of the order of card(V ) � card(K), a constant associated withL(M), independent of the length of w.At the limit, also the complexity of the computation in the proof ofTheorem 5 is constant: for w such thatjwj = 5n(n+ 1)2we use a surface equal to S(jwj) = (8n+ 4)2;hence limjwj!1 S(jwj)jwj = 1285 :A constant is no longer available as a bound of the surface complexity ofrecursively enumerable languages: there is no mapping bounding the lengthof a derivation in a type-0 grammar with respect to the length of the gener-ated string. Thus, there is no mapping bounding the (minimal) surface of arectangle computing a string in a recursively enumerable language. Maybeby imposing such a restriction we can get a characterization of context-sensitive languages (similar to the characterization of context-sensitive lan-guages as languages recognizable in linear space by Turing machines).8 ConclusionsWe have introduced a purely pictorial computability model, the shape gram-mars, which proves to be computationally complete: a characterization ofrecursively enumerable languages is obtained in this framework. Restrictedclasses of shape grammars also prove to be rather powerful. This approachis complementary to approaches present in literature which either de�ne 2Dlanguages via 1D description languages, without capturing the whole rich-ness of the image, or de�ne 2D languages via 2D operations, without relatingthem to 1D languages at all. 22

These results can be seen as a mathematical proof of the intuitive beliefin the expressive power of pictures, of visual languages in general. Thisalso warns about the fact that, due to this power, such languages are highlyundecidable.The surface as a measure of computational complexity can also be apossible tool in checking the intuition that \a larger picture can say morethan a smaller picture" (which is relevant, for instance, with respect to thesize of icons used in man-computer communication).Shape grammars deserve further investigations and we hope to return totheir study.References[1] M. Andries, G. Engels, J. Rekers, How to represent a visual program?,in K. Marriott, B.Meyer eds., Theory of Visual Languages, Springer-Verlag, 1997, to appear.[2] P. Bottoni, M. F. Costabile, S. Levialdi, P. Mussio, Formalizing visuallanguages, in Proceedings of IEEE Symposium on Visual Languages 095,1995, 334 { 341.[3] P. Bottoni, M. F. Costabile, S. Levialdi, P. Mussio, De�ning visuallanguages for interactive computing, IEEE Transactions on Systems,Man, and Cybernetics, 1997 (to appear).[4] P. Bottoni, S. Levialdi, Gh. P�aun, An undecidability result for interac-tive visual languages, submitted, 1997.[5] J. C. Brancheau, C.V.Brown, The Management of End-User Comput-ing: Status and Direction, Computing Surveys, 25(4), 1993.[6] S. K. Chang, M. J. Tauber, B. Yu, J. S. Yu, The SIL-Icon compiler.An icon-oriented system generator, International Journal of PatternRecognition and Arti�cial Intelligence, 2, 2 (1988), 241 { 273.[7] K. Culik II, A purely homomorphic characterization of recursively enu-merable sets, Journal of the ACM, 26 (1979), 345 { 350.[8] M. F. Dacey, The syntax of a triangle and some other �gures , PatternRecognition, 2 (1970), 11 { 31.[9] J. Dassow, Gh. P�aun, Regulated Rewriting in Formal Language Theory,Springer-Verlag, Berlin, Heidelberg, 1989.[10] H. Freeman, On encoding aarbitrary geometric con�gurations, IRETrans. on Electronic Computing, 10 (1961), 260 { 268.23

[11] D. Giammarresi, A. Restivo, Two-dimensional languages, chapter 4 involume 3 of [27].[12] D. Giammarresi, A. Restivo, Recognizable picture languages, in [28],31 { 46.[13] E. J. Golin, Parsing visual languages with picture layout grammars,Journal of Visual Languages and Computing, 2, 4, (1991) 371 { 394.[14] S. J. Guastello, M. Traut, G. Korienek, Verbal versus pictorial repre-sentations of objects in human-computer interfaces, Journal of Man-Machine Studies, 31 (1989), 99 { 120.[15] J. Hartmanis, On the weight of computation, Bulletin EATCS, 55(1995), 136 { 138.[16] D. Hauschild, M. Jantzen, Petri nets algorithms in the theory of matrixgrammars, Acta Inform., 31 (1994), 719 { 728.[17] K. Inoue, I. Takanami, A survey of two-dimensional automata theory,Information Sciences, 55 (1991), 99 { 121.[18] L. Kari, Gh. P�aun, G. Rozenberg, A. Salomaa, S. Yu, DNA computing,sticker systems, and universality, Acta Informatica, to appear.[19] R. A. Kirsch, Computer interpretation of English text and picture pat-terns, IEEE Trans. on Electronic Computers, 13 (1964), 363 { 376.[20] H. A. Maurer, G. Rozenberg, E. Welzl, Using string languages to de-scribe picture languages, Inform. Control, 54 (1982), 155 { 185.[21] P.Mussio, M.Pietrogrande, M.Protti, Simulation of hepatological mod-els: a study in visual interactive exploration of scienti�c problems, Jour-nal of Visual Languages and Computing, 2 (1991), 75 { 95.[22] M. Nivat, A. Saoudi, K. G. Subramanian, R. Siromoney, V. R. Dare,Puzzle grammars and context-free array grammars, Pre-publication990-6, Dept. of Math. and Computer Sci., Univ. Paris-Nord CSP, 1990.[23] A.Repenning, T.Sumner, Agentsheets: a medium for creating domain-oriented visual languages, IEEE Computer, 28, 3 (1995), 7 { 25.[24] A. Rosenfeld, Picture Languages. Formal Models for Picture Recogni-tion, Academic Press, New York, 1979.[25] A. Rosenfeld, R. Siromoney, Picture languages - a survey, Languages ofdesign, 1, 3 (1993), 229 { 245.[26] D. J. Rosenkrantz, Programmed grammars and classes of formal lan-guages, Journal of the ACM, 16, (1969), 107 { 131.24

[27] G. Rozenberg, A. Salomaa, Eds.,Handbook of Formal Languages, 3 vol-umes, Springer-Verlag, Heidelberg, 1997.[28] A. Saoudi, M. Nivat, P. S. P. Wang, Eds., Parallel Image Processing,vol. 4 in Series Machine Perception and Arti�cial Intelligence, WorldSci. Publ., Singapore, 1992.[29] A. Salomaa, Equality sets for homomorphisms of free monoids, ActaCybernetica, 4 (1978) 127-139.[30] A. Salomaa, Jewels of Formal Language Theory, Computer SciencePress, Rockville, Maryland, 1981.[31] A. C. Shaw, The formal picture description scheme as a basis for pictureprocessing systems, Inform. Control 14 (1969), 9 { 52.[32] Y. Yamamoto, K. Morita, K. Sugata, Context-sensitivity of two-dimensional regular array grammars, Intern. J. Pattern Recogn. Artif.Intell., 3, 3-4 (1989), 295 { 319 (and in Array Grammars, Patterns andRecognizers, P. S. P. Wang, ed., World Sci. Publ., Singapore, 1989, 17- 41).[33] P. Wegner, Interaction as a Basis for Empirical Computer Science,Computing Surveys, 27,(1995), 45-48.

25

Turku Centre for Computer ScienceLemmink�aisenkatu 14FIN-20520 TurkuFinlandhttp://www.tucs.abo.�University of Turku� Department of Mathematical Sciences�Abo Akademi University� Department of Computer Science� Institute for Advanced Management Systems ResearchTurku School of Economics and Business Administration� Institute of Information Systems Science