Draft June 10, 2002 688 Distributed Programming (incomplete) might have already edited the object in question, leading to a conﬂict. There are many ways to resolve conﬂicts. F

DraftJune 10, 2002

Part III

Specialized Computation Models

Copyright c© 2001-2 by P. Van Roy and S. Haridi. All rights reserved.

DraftJune 10, 2

002

DraftJune 10, 2

002

Chapter 10

Graphic User InterfaceProgramming (incomplete)

This chapter shows how to define a particularly useful and powerful way to dographic user interface (GUI) programming. We combine the declarative modeltogether with the stateful concurrent model in an approach that gives the bestof both models. To introduce the approach, let us first summarize the existingapproaches:

• Purely procedural. The user interface is constructed by a sequence of graph-ics commands. These commands can be purely imperative, as in tcl/tk,object-oriented, as in the Java AWT (Abstract Window Toolkit) packageor its extension, the Swing components, or even functional, as in Haskellfudgets. The object-oriented or functional style is preferable to an impera-tive style because it is easier to structure the graphics commands.

• Purely declarative. The user interface is constructed by choosing from a setof predefined possibilities. This is an example of descriptive declarativeness,as explained in Section 3.1.

• Using an interface builder. The user interface is constructed manually bythe developer, using a direct manipulation interface.

The purely procedural and declarative approaches have the advantage that theuser interface can be modified at run time. The procedural approach has thedisadvantage that it is complex to use. The declarative approach has the disad-vantage of limited expressiveness. The interface builder approach has the advan-tage of immediate feedback but the disadvantage that the interface is fixed at runtime. None of these approaches is satisfactory. In our view, this is because eachis limited to a single computation model.

This chapter presents a powerful technique for building graphical user inter-faces (GUIs) that combines a declarative base together with the judicious use ofprocedural concepts including objects and threads. This chapter has two goals:


DraftJune 10, 2002

612 Graphic User Interface Programming (incomplete)

• To give a realistic example that shows the advantages of programming withconcepts instead of programming in models. We start from the declarativeprogramming techniques of Chapter 3 and add state and concurrency ex-actly where it is needed. This is a practical example of combining severalcomputation models.

• To present the ideas underlying a practical tool for GUI design that givesthe user a high level of abstraction. It turns out that the combination ofdeclarative and non-declarative (i.e., procedural) techniques is particularlyappropriate to graphic user interface design.

We say a GUI tool is declarative if, insofar as is possible, the GUI is defined by adescription of what it is, instead of by a list of commands that define how to buildit. This is definitional declarativity, as it is introduced in Section 3.1. As we willsee, there are fundamental reasons why the whole interface cannot be defined ina declarative way. Some parts of the GUI must be defined in a procedural way,i.e., by defining how it is implemented.

The ideas in this chapter are embodied in the QTk module, which is part ofthe Mozart system [60]. QTk (“Quick Tk”) is a full-featured GUI design toolbased on the declarative approach [61, 62]. It is not a toy; it has been used tobuild GUIs for real applications. All the examples we give can be run directlywith QTk.

Another desirable property of GUI tools, independent of declarative or pro-cedural specifications, is that the tool be dynamic. By dynamic we mean thatall operations of the tool are possible at run time. This allows building flexibleinterfaces that depend on the state of execution of the application. This propertyhas many desirable consequences:

• With this property it takes just a few lines of code to build an “elastic”widget that changes its form depending on some external condition (e.g.,the window could display a number when it is small and become a graphwhen it gets big enough).

• With this property, different user interface “models” (presentation model,dialog model, etc.) can be represented in the language and transformed toexecutable form directly within the language.

10.1 Declarative and procedural approaches

What are the relative merits of the declarative and procedural approaches tospecifying user interfaces? The trade-off is between manipulability and expres-siveness:

• The declarative approach defines a set of possibilities for different attributes.The developer chooses among this set and defines a data structure that de-


DraftJune 10, 2002

10.1 Declarative and procedural approaches 613

scribes the interface. A purely declarative approach makes it easy to formal-ly manipulate the user interface definitions, e.g., to translate raw data intoa user interface or to change representations. However, the expressivenessis limited because it is only possible to express what the designers initiallythought of.

• The procedural approach gives a set of primitive operations and the abilityto write programs with them. These programs construct the interface. Apurely procedural approach has no limits on expressiveness, since in itsgeneral form it defines a full-fledged programming language. However, thislimits the formal manipulations that can be performed on the user interfacedefinitions.

This trade-off is not a temporary state of affairs, to be solved by some inge-nious new approach. It is a deep property of computation models. As a languagebecomes more expressive, its programs become less amenable to formal manipula-tion. This is illustrated by the Halting Problem.1 However, despite this trade-off,it is still possible to define a model that is both manipulable and expressive. It isa combination of the declarative and procedural approaches. It uses the declar-ative approach in those areas where manipulability is important but a limitedexpressiveness is sufficient. It uses the procedural approach for those areas whereexpressiveness is essential. To be precise, for each window we define four partsdeclaratively:

• The static structure of the window as a set of nested widgets.

• The widget types.

• The initial states of the widgets.

• The resize behavior of the window, i.e., how the widgets change size andrelative position when the window size changes.

We define two parts procedurally:

• Procedures that are executed when external events happen. These proce-dures are called actions. Events are external activities that are detected bythe window.

• Objects that can be called to change the interface in various ways. Theseobjects are called handlers.

The complete definition of the interface is a nested record value with embeddedprocedures and objects. Since it is a record, all the declarative parts can be

1Assume a language as expressive as a Turing machine. Then it is impossible to write aprogram that, given an input program, determines in finite time whether or not the inputprogram will halt.


DraftJune 10, 2

002


formally manipulated. Since it has procedures and objects, it can do arbitrarycomputations.

An important trend in user interface design is to use the declarative approachmore and more. If we understand what expressiveness we really need, then thedeclarative approach is better because it is more concise and easier to manipulate.This becomes practical as we understand how more parts of the interface can beexpressed satisfactorily in this way. To support this trend, the interface modelmust make it easy to build new abstractions on top of it. We provide this withthe data and procedural abstraction ability of the underlying computation model.

10.2 Basic concepts

As much of the interface as possible is defined declaratively as record values.Records are a good choice for two reasons: they are very general data structuresand it is easy to calculate with them. The GUI consists of a set of widgets, whereeach widget is specified by a record. Specifying a GUI is done not by defining anew mini-language, but by using records in the existing language. Programminga complex GUI then becomes a simple matter of doing calculations with records.Since records are strongly supported by the declarative model, these calculationsare easy to specify and efficient.

10.2.1 Basic user interface elements

The GUI model of this chapter has five basic elements:

• Windows and widgets. A window is a rectangular area of the screen thatcontains a set of widgets arranged hierarchically according to a particularlayout. A widget is a GUI primitive that is represented visually on the screenand that contains an interaction protocol, which defines its interactions witha human user. A widget is specified by a record, which gives its type, initialstate, a reference to its handler, and some of the actions it can invoke (seebelow). An interaction protocol defines what information is displayed bythe widget and what sequences of user commands and widget actions areacceptable.

• Events and actions. An event is a well-defined discrete interaction by theexternal world on the user interface. An event is defined by its type, thetime at which it occurs, and possibly some additional information (such asthe mouse coordinates). Events are not seen directly by the program, butonly indirectly by means of actions. An event can trigger the invocation ofan action. An action is a procedure that is invoked when a particular eventoccurs.

• Handlers. A handler is an object with which the program can control awidget. Each widget can have a corresponding handler.


DraftJune 10, 2

002

10.2 Basic concepts 615

User interface description: recordcontaining action procedures and

unbound variables (for handler objects)

Window on screen

Build user interface

Actions and handler objects

(interpret description to createhandler objects and event thread)

Data to be displayed

Calculate user interface description

ApplicationHuman user

Figure 10.1: Building the user interface

10.2.2 Building the user interface

Figure 10.1 shows how the user interface is built. It starts with the data tobe displayed. This data is manipulated to create a record data structure, theuser interface description. This description defines the logical structure of theinterface as a nested record. The record contains embedded action procedures andunbound variables which will become references to handler objects. The record ispassed to a procedure Build , which interprets it and builds the interface. Build

does two things:

• It builds a window on top of an underlying graphics package. The Mozartsystem currently supports two packages: tcl/tk (through the QTk module)and Dynamic HTML (through the QHTMLmodule). This gives the twobuild procedures QTk.build and QHTML.build . Because of the differencesbetween tcl/tk and Dynamic HTML, these procedures have slightly differentfunctionalities. However, most of the basic widgets are the same for both.

• It sets up an internal mechanism so that the application can interact withthe window. The interaction is done with action procedures and handlerobjects. Build creates the handler objects and sets up a thread that be-haves as an event loop. The handler objects are bound to the unboundvariables in the record. The action procedures are executed sequentially inthe thread as window events arrive.


DraftJune 10, 2002


Figure 10.2: Simple text entry window

fun {GetText A}H TD=td(lr(label(text:A)

entry(handle:H))button(text:"Ok"

action: proc {$} T={H get($)} {W close} end ))W={Build D}

in{W show}T

end

Figure 10.3: Function for doing text entry

10.2.3 Example

The easiest way to see how this works is by means of an example. Here is a simpleuser interface description defined as a record:

D=button(text:"Click this button")

The record D defines a widget of button type and the content of the text fieldgives the initial text in the button. For any widget, the record’s label is mappedto the widget type, the features are mapped to widget parameters, and the featurevalues are mapped to the parameter’s initial values.

There exist widgets that can contain other widgets. By using these, thecomplete user interface is a nested record that defines all the widgets and theirlogical organization on the screen. For example, here is a simple interface fordoing text entry (see Figure 10.2):

D=td(lr(label(text:"Type your name:")entry(handle:H))

button(text:"Ok" action: proc {$} {W close} end ))

The td widget organizes its member widgets in top-down fashion. The lr widgetis similar, but left-right. This example has an action, proc {$} {W close}

end , and a handle, H, which we will explain later. At this point, both H and Warestill unbound variables. Create the window by passing D to the Build procedure:

W={Build D}


DraftJune 10, 2002


Figure 10.4: Windows generated with the lr and td widgets

Figure 10.5: Window generated with newline and continue codes

This binds H to the handler object and Wto an object that represents the window.Now we can display the window:

{W show}

Now the user can type text in the window. At any time, the text in the windowcan be read by calling the handler H:

T={H get($)}

This is usually done when the window is closed. To make sure it is done whenthe window is closed, we can put it inside the action procedure. Let us alsoencapsulate the whole user interface in a function called GetText . Figure 10.3shows the resulting code. Calling GetText will wait until the user types a lineof text and then return the text:

{Browse {GetText "Type your name:"}}

10.2.4 Declarative geometry

In addition to the widgets themselves, there are two other aspects of a windowthat are defined declaratively, namely the geometric arrangement of its widgetsand the behavior of its widgets when the window is resized. We describe each inturn. The geometric arrangement of widgets is defined by means of three specialwidgets that can contain other widgets:


DraftJune 10, 2002


• The lr and td widgets arrange their member widgets left-right or top-down.Figure 10.4 shows the two windows that are displayed with the followingtwo commands:

D=lr(label(text:"left")label(text:"center")label(text:"right"))

W1={Build D}{W1 show}

E=td(label(text:"top")label(text:"center")label(text:"down"))

W2={Build E}{W2 show}

• The placeholder widget defines a rectangular area in the window that cancontain any other widget as long as the window exists. The placeholder’scontent can be changed at any time. In the following example, the windowalternatively contains a label and a pushbutton:

placeholder(handle:P)...{P set(label(text:"Hello"))}...{P set(button(text:"World"))}

Calling {P set(D)} is almost the same as calling {Build D} , i.e., it inter-prets the nested record D and creates handler objects, but the visible effectis limited to the rectangular area of the placeholder widget.

• The lr and td widgets support the special codes newline , empty , andcontinue , which allows to organize their member widgets in a grid struc-ture with aligned rows and columns of the same size (see Figure 10.5). Thecode newline makes the subsequent contained widgets jump to a new row(for lr ) or column (for td ). All the widgets in the new row or column arealigned with the widgets in the previous row or column. The empty spe-cial code leaves an empty box the size of a widget. The continue specialcode lets the previous widget span over an additional box. The followingdescription:

lr(button(text:"One" glue:we)button(text:"Two" glue:we)button(text:"Three" glue:we) newlinebutton(text:"Four" glue:we)button(text:"Five" glue:we)button(text:"Six" glue:we) newlinebutton(text:"Seven" glue:we)button(text:"Eight" glue:we)button(text:"Nine" glue:we) newlineempty button(text:"Zero" glue:we) continue)


DraftJune 10, 2002


Ww e

n

s

Figure 10.6: Declarative resize behavior

Figure 10.7: Window generated with the glue parameter

gives the window of Figure 10.5.

10.2.5 Declarative resize behavior

When the size of a window is changed, the interface has to define how the internalwidgets rearrange themselves. This is called the resize behavior. The resizebehavior is dynamic, i.e., it defines a behavior over time. But it is a sufficientlyrestricted kind of dynamic behavior that we can define it using a descriptivedeclarative model.

We define the resize behavior of a widget by an optional glue parameter,whose value can be any combination of n, s , w, and e. The glue parameterplaces constraints on how the widget is placed and how it resizes. As Figure 10.6illustrates, a widget W is always placed inside a rectangular area and has a“natural” size defined by its contents. One can choose that the widget occupiesthe natural size in either direction (horizontally or vertically) or is expanded totake as much space as possible. For the left-right direction, the w value, whenpresent, will attach the widget to the left side (“west”). The same for the e

value (“east”) and the right side. If w and e are present simultaneously, then the


Draft

June 10, 2002


widget is expanded. Otherwise, it takes up just its natural size. For the top-down direction, the n and s values play the same roles (“north” and “south”).For example, the description:

lr(label(text:"Name" glue:w) entry(glue:we) glue:nwe)

gives the window of Figure 10.7.

10.2.6 Dynamic behavior of widgets

The dynamic behavior of widgets is defined by means of action procedures andhandler objects. Taking the example of Section 10.2.3:

declareED=td(lr(label(text:"Type your name:")

entry(handle:E))button(text:"Ok" action:toplevel#close))

W={Build D}{W show}

The action toplevel#close is part of the button; when the button is clickedthen this causes the window to be closed. Generally, actions are zero-argumentprocedures, except that short-cuts are given for a few common actions such asclosing the window. The handle E is an object that allows to control the textentry widget. For example, here’s how to set the value of the widget’s text entryfield:

{E set(text:"Type here")}

Here is how to read and display the value of the field:

{Browse {E get(text:$)}}

Actions can also be attached to events such as mouse clicks:

proc {P} {Browse ´ clicked with third mouse button! ´ } end{E bind(event:"<3>" action:P)}

The event "<3>" means a click of the third mouse button. Attaching it to E

means that the button has to be clicked when the mouse is over E’s widget.

Concurrency is carefully controlled by QTk. Each window has one threadassociated to it. All actions of that window are executed sequentially in thatthread. This guarantees that the order of events within a window is not lost andthat windows operate autonomously. In addition to one thread per window, QTk

has an internal dispatching thread. This guarantees that if an action suspendsindefinitely in a window, then the operation of QTk and its other windows is notaffected.


DraftJune 10, 2002

10.3 Case studies 621

10.3 Case studies

We present three case studies:

• The first builds a simple calendar widget. It is based on a lr widget withgridding ability. It shows the flexibility of the gridding. It also shows howstate can be introduced to optimize execution of what is originally a purelydeclarative calculation.

• The second derives two different GUIs by transforming one data model intotwo GUI descriptions. This shows the advantage of tightly integrating theGUI tool with the language, since different data models can be representedwith the same language data structures (e.g., records) and transformed withthe same language operations.

• The third defines a clock with an interface that adapts itself according to anexternal condition. The “best view” of the clock data is chosen dynamicallydepending on the window size. Because of the mixed declarative/proceduralapproach, each view is completely defined in just a few lines of code.

All three case studies were written by Donatien Grolaux.

10.3.1 A simple calendar widget

The grid structure of Section 10.2.4 can be used to build widgets with dataarranged in rectangular form. We show how by building a simple calendar widget.Figure 10.8 shows what it looks like. The calendar widget has two importantparts: a widget, to display the calendar, and a redraw procedure, to updatethe date. We define a function {Calendar Redraw} that returns the calendarwidget and binds Redraw to the redraw procedure. The calendar widget is justa placeholder:

declare PCal=placeholder(handle:P)

Let us now define the redraw procedure. It is a one-argument procedure {Redraw

T} called with a time argument T in a record format. For simplicity, let usredraw the complete calendar each time. Assume the time has the format of the{OS.localTime} call, which looks like this:

time(hour:11 isDst:0 mDay:12 min:5 mon:11 sec:7 wDay:2yDay:346 year:100)

For the calendar we need only the fields wDay (weekday, 0 to 6, where 0 is Sunday),mDay(day of month, 1 to 31), mon (month, 0 to 11), and year (years since 1900).The redraw procedure is called regularly to update the calendar.

The calendar is a rectangular grid of weekday names and day numbers. Webuild this grid by using an lr widget with its gridding ability. We list all the


DraftJune 10, 2002


Figure 10.8: A simple calendar widget

calendar elements and use newline to go the next line in the calendar. We startby defining the calendar’s header:

Header=[label newlinelabel(text:"Mo") label(text:"Tu") label(text:"We")label(text:"Th") label(text:"Fr") label(text:"Sa")label(text:"Su") newlinelrline(glue:we) continue continue continue continuecontinue continue newline]

This displays the weekday names and underlines them. We now make a list thatcontains all calendar numbers. First, we calculate the number of days in themonth, taking leap years into account:

ML={List.nth [31if (T.year div 4)==0 then 29 else 28 end31 30 31 30 31 31 30 31 30 31] T.mon+1}

Second, we calculate the number of blank spots in the grid before the day number“1”:

SD=(((7-(T.mDay mod 7))+T.wDay) mod 7)

With these two numbers, we can make a list of calendar days, correctly offset inthe month:

fun {Loop Skip Col Nu}if Nu>ML then nilelseif Col==8 then

newline|{Loop Skip 1 Nu}elseif Skip>0 then

label|{Loop Skip-1 Col+1 Nu}elseif Nu==T.mDay then

label(text:Nu bg:black fg:white)|{Loop 0 Col+1 Nu+1}


DraftJune 10, 2

002


elselabel(text:Nu)|{Loop 0 Col+1 Nu+1}

endendR={List.append Header {Loop SD 1 1}}

Finally, we create the calendar and put it in the placeholder:

{P set({Record.adjoin lr {List.toTuple lr R}})}

Memoization: using state to avoid repeating work

This redraw procedure will redraw the whole calendar each time it is called. Formany clocks this will be once per second or once per minute. This is very wastefulof computational resources. We can avoid the repetitive redraw by storing theyDay , year , and mon fields together in a cell, and redrawing only if the contentchanges:

fun {Calendar Redraw}Date={NewCell r(yDay:0 year:0 mon:0)}

inproc {Redraw T}

TOld=@DateTNew=r(yDay:T.yDay year:T.year mon:T.mon)

inif TOld==TNew then skipelse

... % Recalculate calendar as beforeDate:=TNew

endend...

end

The unoptimized redraw procedure has a declarative implementation and theoptimized redraw procedure has a stateful implementation. Yet, both have iden-tical, declarative behavior when viewed from the outside. The state is used justto memorize the previous calendar calculation so that the procedure can avoiddoing the same calculation twice. Paraphrasing philosopher George Santayana,we can say this is “remembering the past to avoid repeating it”. This techniqueis called memoization. It is a common use of state. It is particularly nice be-cause it is modular: the use of state is limited to the inside of the procedure (seeSection 5.7.2).

10.3.2 Automatic generation of user interfaces

Using records to specify the user interface makes it possible to calculate the us-er interface directly from the data. This is a powerful technique that supports


DraftJune 10, 2

002


Figure 10.9: Automatic generation of user interfaces

advanced approaches to GUI design. One such approach is model-based design.In this approach, different aspects of a GUI design are specified in separate for-malisms called models. A running GUI is then obtained from all the models takentogether [117]. Typical models include a domain model, a presentation model, adialog model, a task model, a user model, a platform model, and a help model.These models can be represented within the language as data structures consist-ing of nested records and procedure values. From the records, it is possible tocalculate QTk records. It is particularly easy to translate the domain model intothe presentation and dialog models, because the latter are directly supported byQTk.

The domain model is very similar to the application’s data model. It definesthe data entities that the user can manipulate. The presentation model is arepresentation of the visual, auditive and other elements that the UI offers tothe users. The dialog model defines how the presentation model interacts withthe user. It represents the actions that a user can initiate via the presentationelements and the responses of the application.

We give an example to show how one domain model can be automaticallymapped to two presentation models: a read-only view and an editable view of thedata. The user can freely switch between the views by clicking a check button (seeFigure 10.9). The mapping from the domain model to the presentation modelsis done in a natural way by means of functions, using declarative programming.There are two functions, ViewPresentation and EditPresentation , each ofwhich calculates the QTk presentation model from the same original data (seeFigure 10.10). The presentation model is encapsulated in a common interface forthe two views. A placeholder is used to dynamically display one of the two views.Because of the common interface, keeping coherence between views is trivial.

We give the commented source code of the example. We use a very simpledomain model: a list of pairs of the form identifier#value , which representsthe known information about some entity. The purpose of the GUI is to displayor edit this information. Let us take the following information to start with:

Data=[name#"Roger"


Draft

June 10, 2002


Original data

UIEditable mode Read-only mode

widget (QTk)Placeholder

UI

EditPresentation ViewPresentation

Figure 10.10: From the original data to the user interface

surname#"Rabbit"age#14]

Now we define the two functions that translate this domain representation intothe view information needed by QTk. The first function, ViewPresentation ,builds the read-only view. It builds a representation where each pair identifi-

er#value is mapped to a label widget whose text is the identifier followed by a”:” and the corresponding value. The function returns a record with four fields:

• Field desc : the description record, which describes the widget in QTk for-mat.

• Field handle : will be bound to the widget’s handle.

• Field set : a one-argument procedure that updates the information dis-played by the widget.

• Field get : a function that returns the displayed information. For simplicity,this returns the last information that was set.

Here is the function definition:

fun {ViewPresentation Data}Handle In={NewCell Data}

inr(desc: {Record.adjoin

{List.toTuple td {List.map Datafun {$ D#V}

label(glue:we feature:D text:D#":"#V) end }}td(glue:nswe handle:Handle)}

handle: Handleset: proc {$ N}


DraftJune 10, 2002


In:=N{ForAll N

proc {$ D#V} {Handle.D set(text:D#":"#V)} end }end

get: fun {$} @In end )end

The second function, EditPresentation , builds the editable view. It builds arepresentation where each pair identifier#value is mapped to a label con-taining the identifier followed by ”:” and an entry widget (if the value is a string)or a numberentry widget (if the value is an integer). This representation usesthe gridded structure. This function returns a record with four fields, in similarfashion to the ViewPresentation function. This time, the result of the get

function is obtained from the widgets themselves.

fun {EditPresentation Data}Handle Feats={List.map Data fun {$ D#_} D end }fun {Loop X}

caseX of D#V|Xs then

label(glue:e text:D#":") |if {IsInt V} then numberentry(feature:D init:V glue:we)else entry(feature:D init:V glue:we) end |newline | {Loop Xs}

else nil endend

inr(desc: {Record.adjoin

{List.toTuple lr {Loop Data}}lr(glue:nswe handle:Handle)}

handle: Handleset: proc {$ N}

{ForAll N proc {$ D#V} {Handle.D set(V)} end }end

get: fun {$} {List.map Featsfun {$ D} D#{Handle.D get($)} end } end )

end

The main application calls both of these functions on the same data. Themain window contains a placeholder widget and a checkbox widget. Checkingor unchecking the box switches between both views, maintaining data integritybetween them by using their associated set and get operations. Once the win-dow is built, the descriptions of both views are put in the placeholder widget.Thereafter they can be placed back at any time by using their handles. Here isthe main application:

V1={ViewPresentation Data}V2={EditPresentation Data}P C


DraftJune 10, 2

002


{{Buildtd(placeholder(glue:nswe handle:P)

checkbutton(text:"Edit" init: false handle:Caction:

proc {$}Old#New= if {C get($)} then V1#V2 else V2#V1 endV={Old.get}

in{New.set V}{P set(New.handle)}

end ))}show}

{P set(V2.desc)}{P set(V1.desc)}

This example shows the advantages of tightly integrating an executable model-based GUI with an expressive programming language. The different models canall be expressed within the language. The mappings between the different modelscan then be done easily within the language.

10.3.3 A context-sensitive clock

This section defines a simple clock utility, FlexClock, that dynamically displays adifferent view depending on the size of the window. When the window is resized,the “best” view for the new size is chosen. This application has six views andneeds less than 100 lines of code. It dynamically chooses the best view amongthe six. The best view is the one that gives the most detailed time informationin the available window size. Figure 10.12 shows three of the six views. A moreelaborate version, with 16 views, an analog clock widget, and a calendar widget,is available as part of the Mozart demos.

Figure 10.11 shows the architecture of the clock. Each view consists of threeparts: an update procedure, a widget definition, and a minimum size. A clockcalls all the update procedures once a second with the current time. A viewselector picks one of the views and puts it in a placeholder widget to displayit. Each time the window is resized, i.e., whenever a resize event occurs, thenthe view selector chooses the best view and displays it. The update proceduresand the view selector run concurrently. This is acceptable because there is nointerference between them.

The following code sets up a small server that periodically calls each procedurein a list of procedures:

NotifyClocklocal

Clocks={NewCell nil}proc {Loop}

T={OS.localTime}in


DraftJune 10, 2002


...(does updates oncea second with time)

procedureUpdate Widget

(record)Minimum

size

(each definition has three parts)

Definition of views

PlaceholderselectorView

widget

eventsWindow resize

Clock

(uses minimum size)

Figure 10.11: Architecture of the context-sensitive clock

for I in @Clocks do {I T} end{Delay 1000}{Loop}

endin

proc {NotifyClock P}Clocks:=P|@Clocks

endthread {Loop} end

end

The period is almost exactly once per second.2 Calling {NotifyClock P} addsP to the list of procedures to be notified. The update procedure of each viewwill be put in this list. The OS.localTime call returns a record that containsall the time information. To help in calculating the views, we define some utilityfunctions to format the time information in various ways:

fun {TwoPos I} if I<10 then "0"#I else I end endfun {FmtTime T} {TwoPos T.hour}#":"#{TwoPos T.min} endfun {FmtTimeS T} {FmtTime T}#":"#{TwoPos T.sec} endfun {FmtDate T} {TwoPos T.mDay}#"/"#{TwoPos T.mon+1} endfun {FmtDateY T} {FmtDate T}#"/"#(1900+T.year) endfun {FmtDay T}

{List.nth ["Sunday" "Monday" "Tuesday" "Wednesday""Thursday" "Friday" "Saturday"] T.wDay+1}

endfun {FmtMonth T}

{List.nth ["January" "February" "March" "April" "May""June" "July" "August" "September" "October"

2Section 4.6.2 explains how to do it exactly once per second.


DraftJune 10, 2002


Figure 10.12: Three views of FlexClock

"November" "December"]T.mon+1}

end

Now we define each view as a record with three elements:

• A declarative description of the widgets comprising the view.

• A procedure that when called updates the displayed information accordingto the current time.

• The minimal size (horizontally and vertically) that is required to correctlydisplay the view.

Figure 10.13 defines all six views in one list. The window that displays one ofthese views is simple: it contains a single placeholder widget. A placeholder, aswe saw before, is a container that can contain any widget and that can changethe displayed widget at any time as long as the window exists.

PWindow={Build

td(title:"FlexClock demo"placeholder(handle:P width:1 height:1 glue:nswe))}

To initialize the application, all views are placed once in the placeholder. Afterthis is done, any view can be displayed again in the placeholder by a singlecommand using the view’s handle. We also register each update procedure withNotifyClock . The result is a list, Views , that has one triple per view, containingthe minimal width, minimal height, and handle for the view:

Views={List.map ViewListfun {$ R}

Width#Height=R.surface


DraftJune 10, 2

002


declareH1 H2 H3 H4 H5 H6ViewList=

[r(refresh: proc {$ T} {H1 set(text:{FmtTime T})} enddesc:label(handle:H1 glue:nswe bg:white)surface:40#10)

r(refresh: proc {$ T} {H2 set(text:{FmtTimeS T})} enddesc:label(handle:H2 glue:nswe bg:white)surface:80#10)

r(refresh:proc {$ T}

{H3 set(text:{FmtTime T}# ´ \n ´ #{FmtDate T})} enddesc:label(handle:H3 glue:nswe bg:white)surface:40#30)


{H4 set(text:{FmtTimeS T}# ´ \n ´ #{FmtDateY T})}end

desc:label(handle:H4 glue:nswe bg:white)surface:80#30)


{H5 set(text:{FmtTimeS T}# ´ \n ´ #{FmtDay T}#", "#{FmtDateY T})} end

desc:label(handle:H5 glue:nswe bg:white)surface:130#30)


{H6 set(text:{FmtTimeS T}# ´ \n ´ #{FmtDay T}#", "#T.mDay#" "#{FmtMonth T}#" "#(1900+T.year))}

enddesc:label(handle:H6 glue:nswe bg:white)surface:180#30)]

Figure 10.13: View definitions for context-sensitive clock


DraftJune 10, 2

002


Minimal size for view 3

Area in which view 3 is chosen(3)

3

1501005000

6543

1 2

(1)(1) (2) (2) (2)

Width

40 (1) (3) (4) (5) (6)

Height

Figure 10.14: The best view for any size clock window

in{P set(R.desc)}{NotifyClock R.refresh}Width#Height#(R.desc).handle

end }

Now we have initialized the placeholder and registered all the update procedures.The next step is to set up the mechanism to calculate the best view and displayit. The best view is the one that satisfies the following three conditions:

• The window size is big enough to display the view, i.e., window width ≥minimal view width and window height ≥ minimal view height.

• The distance between the bottom right corner of the minimal view and thebottom right corner of the window is minimized. It is sufficient to minimizethe square of this distance.

• If no view satisfies the above two conditions, the smallest view is chosen bydefault.

Figure 10.14 shows how this divides up the plane among the six views. The pro-cedure Place does the calculation and displays the best view in the placeholder:

proc {Place}WW={QTk.wInfo width(P)}WH={QTk.wInfo height(P)}_#Handle={List.foldRInd Views

fun {$ I W#H#Handle Min#CH}This=(W-WW)*(W-WW)+(H-WH)*(H-WH)

inif W<WWandthen H<WHandthen

(Min==inf orelse This<Min) then This#Handleelse Min#CH end


DraftJune 10, 2002


endinf# local (_#_#H)|_=Views in H end }

in{P set(Handle)}

end

This starts with a minimum of inf , representing infinity, and is reduced for eachview with a smaller distance. When the window is resized, Place has to be calledto set the correct view according to the new size of the window. This is done bybinding the <Configure> event of QTk:

{P bind(event: ´ <Configure> ´ action:Place)}

The window is now completely set up, but it is still hidden. We display it asfollows:

{Window show}{Window wait}

10.4 Implementing the GUI tool

The tool has two main parts: an interpreter (the QTk module) for the declarativespecification and an interface (the Tk module) to the underlying graphics package.When a specification is given to the interpreter, it builds the user interface bygiving commands to the graphics package, using an almost-purely proceduralmodel. The interpretation only has to be done once; afterwards the only overheadis the type checking of handler arguments.

The placeholder widget deserves mention because new widgets can be addedto it at any time. It contains an arbitrary number of other widgets. These otherwidgets are interpreted just once, when put in the placeholder; changing whichone is displayed has no interpretation overhead. This is true for each new widgetadded to the placeholder, including those added during execution.

10.5 Exercises

1. Calendar widget. Section 10.3.1 gives a formula for calculating the num-ber of blank spots in the grid before day 1. Why is this formula correct?

2. Clock with calendar. Extend the context-sensitive clock of Section 10.3with a calendar and add views so that the calendar is shown only when thewindow is big enough. You can use the calendar widget defined in the samesection.


Draft

June 10, 2002

Chapter 11

Distributed Programming(incomplete)

A distributed system is a set of autonomous processes linked together by a net-work [157, 98, 31]. To emphasize that these processes are not necessarily on thesame machine, we call them sites. Sites have different address spaces and differentsets of local resources. From the viewpoint of an application, the computation ispartitioned between the sites. The sites communicate among one another to dothe computation.

Programming a distributed system can be quite hard if the programmer has totake into account all possible interactions between the sites. We have found that agood approach to make things simpler is to hide the distributed operations insidethe basic language operations. This is called network-transparent distribution.This idea has a long history. The first implementation was the Remote ProcedureCall (RPC), done in the early 1980’s [16]. It allows to call a remote procedure inalmost the same way as a local procedure. Recently, the idea has been extendedto object-oriented programming by allowing methods to be invoked remotely.This is called Remote Method Invocation (RMI). This technique has been madepopular by the Java programming language [154]. This chapter pushes the ideafarther in two ways:

• All language entities, not just procedures (RPC) and objects (RMI), aregiven a well-defined distribution behavior.

• Operations on the language entities have exactly the same language behaviorlocally and remotely.

In other words, a program does the same computation no matter whether it runson one site or is spread out over many sites. From the programmer’s point ofview, the network is invisible. Whether a computation happens here or theredoes not affect what the program does. It only affects how long it takes.


DraftJune 10, 2002

634 Distributed Programming (incomplete)

Part of problem

Interaction between parts

Distributionstructure

Applicationfunctionality

Applicationfunctionality

Resource use

Fault tolerance

Security

Distribution structure

Parallelism

These have small, well-definedeffects on functionality

with added specificationsSingle model

Resource use

SecurityParallelism

Faulttolerance

Multiple interacting models

Figure 11.1: The challenge: simplifying distributed programming

Transparency is not enough

It is not so difficult to achieve network transparency. It is enough to implementthe basic language operations in a distributed way, so that their semantics ismaintained independent of distribution structure. However, just achieving net-work transparency is far from sufficient for practical distributed programming.There are many other concerns:

• Distribution structure. Partitioning the computation changes executiontiming. For example, network delays change timing of the basic languageoperations. Threads running on different sites will be scheduled differentlythan those running on the same site. Applications should be aware of theseeffects and have some control over them.

• Localized resources. Each site has resources, i.e., modules that can onlybe executed on that site. For example, each site has its own GUI modulethat accesses its own keyboard, mouse, and screen. Each site has its ownlocal file system and local peripherals such as printers and scanners. Appli-cations that depend on resources have to decide which site’s resources theyuse.

• Partial failure. In a centralized system, failure is all-or-nothing: the wholeprocess either runs correctly or it is failed. In a distributed system this isdifferent: it can exhibit partial failure, in which just part of the systemfails. For example, each site can crash independently of other sites and thenetwork can become temporarily inaccessible between any two sites. Anapplication running on just one site does not have to take this into account,but if it is distributed, then this becomes important.


DraftJune 10, 2

002

635

• Security. Applications can share the same physical infrastructure, i.e.,they communicate over the same network. How do we keep them frominterfering with each other? An application can exist over several trustdomains, where a trust domain is a subset of the physical infrastructureinside of which different computations trust one another. A single site isinside one trust domain. How does the application manage these differentlevels of trust?

The above properties make it harder for applications to satisfy users. There aretwo additional properties that can make some things easier:

• Redundancy. A distributed system is redundant, i.e., there is duplicationof resources. For example, there can be many sites running simultaneously.Redundancy and partial failure are two sides of the same coin. On theone hand, a redundant system can partially fail. On the other hand, theapplication can take advantage of redundancy to compensate for partialfailure. It can duplicate part of its execution state so that it can continueto work correctly even if part of the system fails. We say that the programis fault tolerant. What’s more, redundancy can be a positive boon: it canincrease reliability and availability beyond what a centralized system canprovide.

• Parallelism. Multiple sites running on different machines have higher ag-gregate performance than just one machine. Applications can take advan-tage of this parallelism to run faster.

How can we take all these concerns into account without making distributedprogramming too complicated? This is one of the major challenges in distributedprogramming research today. Figure 11.1 illustrates the conceptual problem: howcan we untangle these concerns so that each of them can be treated separately?(The figure combines partial failure and redundancy under the single heading“fault tolerance”.)

Progress in this area is difficult because it requires combined research in pro-gramming languages, system design, and distributed algorithms. This chaptershows one promising approach, in which the computation model is part of the so-lution instead of part of the problem. That is, the computation model is designedto make it easier to address these concerns.

Due to its layered structure, i.e., the clean distinction between stateless, single-assignment, and stateful concepts, we have found the stateful concurrent modela good starting point. With this model as the foundation, we can make progressin separating distribution structure, resource use, and fault tolerance from theapplication functionality.

For a small but important class of performance-critical applications, namelyconstraint problems, it is possible to successfully separate application functional-ity from parallelism (see Chapter 12) [136].


DraftJune 10, 2002


Distributed memorymultiprocessor

Shared memorymultiprocessor

multiprocessorDistributed memory

with partial failure

Open distributed system

(name server issues)(failure and security issues)

Add distribution

Add partial failure

Add openness

Collaboration(‘‘Internet computing’’)

High performance(‘‘cluster computing’’)

?

Figure 11.2: A simple taxonomy of distributed systems

Taxonomy of distributed systems

The computation model of this chapter covers but one type of distributed system.Let us situate it with respect to other kinds of distributed system. To explainthis, Figure 11.2 gives a simple taxonomy of distributed systems. The figureshows four types of distributed system. For each type, there is a simple diagramto illustrate it. In these diagrams, circles are processors, the rectangle is memory,and connecting lines are communication links (a network). The figure startswith a shared-memory multiprocessor, which is a computer with more than oneprocessor attached to a common memory. Communication between processors isextremely fast; it suffices for one processor to write a memory cell and another toread it. Coordinating the processors, so that, e.g., they all agree to do the sameoperation at the same time, is efficient.

When the distance between processors gets large, i.e., much larger than thetime a signal travels during one memory operation, then communication delaysbecome significant. The processors are loosely coupled. We speak of a distributedmemory multiprocessor. Coordinating the processors becomes time-consumingsince it requires all processors to communicate with each other. What’s more, aprocessor can fail even though its siblings are still running. This is called partialfailure. In the figure, a failed processor is a circle crossed with a large X. This leadsto distributed memory multiprocessors with partial failure. Such machines areusually called clusters. They are designed to keep running, albeit with degradedperformance, even if some processors are failed. That is, the probability that thecluster continues to provide its service is close to 1 even if part of the cluster isfailed. This property is called high availability. The purpose of a cluster is toprovide high performance with high availability.


DraftJune 10, 2002

11.1 The distribution model (part I) 637

The last step adds openness, in which independent computations or computerscan find each other, connect, and collaborate meaningfully. This is the modelsupported by the Mozart system and explained in this chapter. Openness is thecrucial difference between the world of high-performance computing and the worldof collaborative computing. In addition to partial failure, openness introduces twonew issues: naming and security. Naming is how computations or computers findeach other. Naming is usually supported by a special part of the system calledthe name server. Security is how computations or computers protect themselvesfrom each other.

Structure of the chapter

This chapter gives a snapshot of ongoing research. Fault tolerance and security,in particular, are still very much active research topics. The chapter has twomain goals. First, it defines a simple, yet useful distributed computation model.Second, it shows how to write robust distributed programs in this model. Thisgives the following structure:

• Section 11.1 introduces and justifies the basic distributed computation mod-el, which extends the stateful concurrent model of Chapter 7.

• Section 11.2 changes gears by giving hands-on examples to give intuitionon the basic concepts. After reading this section, you will be able to writesignificant distributed programs.

• Sections 11.3 and 11.4 then continue where Section 11.1 left off, to give adeeper view of the basic concepts.

• Sections 11.5 and 11.6 give more substantial abstractions for distributedprogramming. Section 11.5 presents a generic tool for building robustclient/server applications. Section 11.6 gives a useful abstraction for ac-tive fault tolerance, namely a transactional object store.

11.1 The distribution model (part I)

We refine the stateful concurrent model of Chapter 7 with a distribution model.This model defines the network operations done for language entities when theyare shared between sites [65, 164, 66, 167, 67]. If distribution is disregarded (i.e.,we do not care how the computation is spread over sites) and there are no failures,then the computation model of the language is the same as if it executes on onesite. There are three issues:

• The first issue is transparency. We would like to keep as much as possi-ble the semantics of the language without distribution. This means thatcentralized programs continue to run and that centralized reasoning is still


DraftJune 10, 2

002


Immutable store Mutable store

b

a

Values are not localized

(dataflow variables and values) (cells)

n

●

● Threads, cells, and dataflowvariables are localized tohome sites {a, b, ..., n, ...}

Threads...

b

(ST1) (ST2) (STn)a b n

(X)

U=c2

(c1:W)(c2:Z)

Z=person(age: Y)

W=atom(V)

Y=c1

Figure 11.3: The distributed computation model

valid. Language entities have two semantics: their language semantics asbefore, plus a new distributed semantics that defines the network behavior.

• The second issue is efficiency. The extended model should be efficient:there should be few network operations. Local operations should require nonetwork operations and others should require minimal network operations.For example, binding a dataflow variable should require one message to eachsite that references it. This means that to a first approximation, distributionbehavior can be ignored.

• The third issue is programmability. The extended model should allow tuningprograms to improve network behavior. The distributed semantics of thelanguage entities should be useful as building blocks for creating exactly thenetwork behavior that is desired. This property is called network awareness.

The extension that we found works well adds just one concept: the notion of site.Computations are partitioned over a set of sites. We assume that any site canhold a reference to a language entity on any other site. Conceptually, there is asingle global computation model that encompasses all running Mozart programsand Mozart data (even those programs that are not connected together). Theglobal store is the union of all the local stores. Figure 11.3 shows the computationmodel.

To add distribution to the global view, the idea is to annotate language entitieswith sites. The distribution behavior (in terms of network operations) dependson the sites. How do we define this behavior? We do it by distinguishing betweenstateful, stateless, and single-assignment entities. Each of them has a differentdistribution behavior:

• Stateful entities (threads, cells, ports, objects) have an internal state. Thedistribution behavior has to be careful to maintain a globally coherent view


DraftJune 10, 2002

11.1 The distribution model (part I) 639

... SnS2S3

S1X

Threads

Cell

c1:X

Dataflow variable

c2:Z Z

W

SxY

Site a Site b Site n

Figure 11.4: Site-oriented view of the distribution model

of the state. This puts major constraints on the kinds of efficient behaviorthat are possible. The simplest kind of behavior is to make them stationary.

• Single-assignment entities (dataflow variables, streams) have one essentialoperation, namely binding. This operation is coordinated from the site onwhich the variable is created.

• Stateless entities, i.e., values (procedures, functions, records, classes, func-tors) do not need a site annotation because they are constants. They canbe copied between sites.

Figure 11.4 shows a set of sites with localized threads, cells, and unbound dataflowvariables. In the stateful concurrent model, the other entities are defined interms of these. For all entities except for ports, the distributed behaviors of thedefined entities can be seen as derived from their parts. Ports are different. Wedefine ports to have an asynchronous FIFO behavior in a distributed setting (seeSection 7.3.2). This distributed behavior does not follow from the definition ofports in terms of cells. This effectively means that ports are basic entities in thedistribution model, just like cells.

This basic model is sufficient to express many useful distributed programs,but it has two limitations: network behavior cannot be tuned precisely enoughin some cases and partial failures are not taken into account. Later on in thischapter, we extend the basic model in two ways to overcome these limitations:

• Some language entities have a more sophisticated distributed semanticsthan just being localized to one site. For example, “cached” (or “mobile”)objects move to the site that is using them. These objects have the samelanguage semantics but a different distributed semantics. This is importantfor tuning network performance.

• The distributed system can have partial failures, i.e., some sites and net-works can fail to operate as they should. The bigger the system, the morelikely that there is a partial failure somewhere. Therefore, the basic modelshould take this into account.


DraftJune 10, 2002


11.2 Hands-on introduction

After this long prelude, let us get down to business in a completely practical waywith hands-on examples. New users typically find it hard to believe that networktransparency does not result in inefficient network operations. We find that usingexamples is the most convincing way to show that the approach is practical. Inthe next few pages, we give a hands-on introduction to the distribution model thatintroduces all the concepts. All these examples can be executed in the Mozartinteractive development environment. Some of the examples use the utilitiesdefined in Appendix B.5.

11.2.1 Basic operations

Network transparency and open distribution

Network transparency means that a program will perform exactly the same com-putation independently of how it is partitioned over a set of sites. This meansthat a language entity used on one site has to behave in rigorously the sameway if it is referenced from many sites. To illustrate this, we give self-containedcode examples that can be run in Mozart’s interactive user interface. In theseexamples, we will reference language entities from two sites.

Open distribution, tickets, and the Connection module

We say a distributed computation is open if it can connect with independently-running distributed computations at run time, without necessarily knowing be-forehand which computations it may connect with nor the type of information itmay exchange. A distributed computation is closed if it knows at all times whichcomputations it will connect with.

To make open distribution possible, we need to have a way for independentcomputations to find each other, and once found, to connect to each other. Oneof the simplest ways for computations to connect to each other is by means oftickets. A ticket is a global reference into a store on a site, represented as anASCII string. Any computation can create a ticket for any local reference. Anyother computation can get the reference by getting the ticket. Since the ticket isan ASCII string, there are a million and one ways that a second process can getthe ticket. It could be through a shared file, through the Web, through email,etc. Later on, we will show how to use the Web for this.

The Connection module Tickets are created and used with the Connection

module. This module has three basic operations:

• {Connection.offer X T} creates a ticket T for any reference X. The ticketcan be taken just once. Attempting to take a ticket more than once willraise an exception.


DraftJune 10, 2002

11.2 Hands-on introduction 641

• {Connection.offerUnlimited X T} creates a ticket T for any referenceX. The ticket can be taken any number of times.

• {Connection.take T X} creates a reference X when given a valid ticket inT. The X refers to exactly the same language entity as the original referencethat was offered when the ticket was created. A ticket can be taken at anysite. If taken at a different site than where the ticket was offered, then thereis network communication between the two sites.

With Connection , connecting computations in different processes is extreme-ly simple. The system does a great deal of work to give this simple view. Itimplements the connection protocol, transparent marshaling and unmarshaling,distributed garbage collection, and a carefully-designed protocol for each languageentity.

Sharing language entities

Sharing records We start with a simple example. The first process has a bigdata structure, a record, that it wants to share. It first creates the ticket:1

X=the_novel(text:"It was a dark and stormy night. ..."author:"E.G.E. Bulwer-Lytton"year:1803)

{Show {Connection.offerUnlimited X}}

Tickets are implemented in the module Connection . This example creates theticket with Connection.offerUnlimited and displays it in the Mozart emula-tor window (with Show). Any other process that wants to get a reference to X

just has to know the ticket. Here is what the other process does:

X2={Connection.take ´ ...ticket comes here... ´ }

(To make this work, you have to replace the text ´ ....ticket comes here.... ´

by what was displayed by the first process.) That’s it. The operation Connec-

tion.take takes the ticket and returns a language reference, which we put inX2. Because of network transparency, both X and X2 behave identically.

Sharing functions This works for other data types as well. Assume the firstprocess has a function instead of a record:

fun {MyEncoder X} (X*4449+1234) mod 33667 end{Show {Connection.offerUnlimited MyEncoder}}

The second process can get the function easily:

E2={Connection.take ´ ...MyEncoders ticket... ´ }{Show {E2 10000}} % Call the function

1Here, as in the subsequent examples, we leave out declare for brevity, but we keepdeclare ... in for clarity.


DraftJune 10, 2002


Sharing dataflow variables In addition to records and functions, the ticketcan also be used to pass unbound variables. Any operation that needs the valuewill wait, even across the network. This is how we do distributed synchroniza-tion [65]. The first process creates the variable and makes it globally accessible:

declare X in{Show {Connection.offerUnlimited X}}

But the process does not bind the variable yet. Other processes can get a referenceto the variable:

declare X inX={Connection.take ´ ...Xs ticket... ´ }{Browse X*X}

The multiplication blocks until X is bound. Try doing X=111 in the first process.The binding will become visible on all sites that reference the variable.

Sharing state The simplest way to share state between processes is by sharinga cell. This can be done exactly in the same way as for the other types.

declareC={NewCell unit }{Show {Connection.offerUnlimited C}}

Any other process can access C by doing:

declareC1={Connection.take ´ ...Cs ticket... ´ }

Any process that references the cell can do exchange operations on it. The systemguarantees that this is globally coherent. I.e., if process 1 first puts foo in thecell, and then process 2 does an exchange, then process 2 will see foo . Knowinghow the system does this is important for efficiency (e.g., network hops). We willsee later how the global coherence is maintained and how the programmer cancontrol what the network operations are.

Sharing objects and other data types A more sophisticated way to sharestate is to share objects. In this way, we encapsulate the shared state and controlwhat the possible operations on it are. Here is how to distribute an object:

class Coderattr seedmeth init(S) seed:=S endmeth get(X)

X=@seedseed:=(@seed*1234+4449) mod 33667

endendC={New Coder init(100)}{Show {Connection.offerUnlimited C}}


DraftJune 10, 2

002


This defines the class Coder and an object C. Any process that takes the object’sticket will reference it. The Mozart system guarantees that the object will behaveexactly like a centralized object. For example, if the object raises an exception,then the exception will be raised in the thread calling the object.

At this point, we have shown how to distribute some language entities: records,functions, dataflow variables, cells, and objects. In fact, we have just scratchedthe surface of what we can distribute with tickets. We can distribute almost anydata type of the language, including procedures, classes, functors, and ports.

Distributed lexical scoping

One of the important properties of network transparency is distributed lexicalscoping: a procedure value that references a language entity will continue toreference that entity, independent of how the procedure value is transferred acrossthe network. This causes remote references to be created implicitly, by the simpleact of copying the procedure value from one site to another. For example:

declareC={NewCell 0}fun {Inc X} X+@C end{Show {Connection.offerUnlimited C}}{Show {Connection.offerUnlimited Inc}}

Inc will always reference C, on any site where it is called. A third site can takeC’s ticket and change the content. This will change the behavior of Inc . Let usgo over this again carefully. Site 1 defines C and Inc . Site 2 gets a referenceto Inc and calls it. Site 3 gets a reference to C and changes its content. Thenwhen site 2 calls Inc again it will use the new content of C. Semantically, thisbehavior is nothing special: it is a consequence of using procedure values withnetwork transparency. But how is it implemented? In particular, what networkoperations are done to guarantee it? We would like the network behavior to besimple and predictable. Fortunately, we can design the system so that this isindeed the case, as Section 11.2.2 explains.

The importance of distributed lexical scoping was first recognized by LucaCardelli and realized in Obliq, a simple functional language with object-basedextensions that was designed for experimenting with network-transparent distri-bution [26].

Ticket distribution, URLs, and the Pickle module

In the above examples we copied and pasted tickets between windows of theinteractive environment. This is not only cumbersome, but it only works on onemachine. A better way is to make tickets available through a file. To successfullyconnect with the ticket, the destination process has to see the file. This notonly makes the distribution easier, it also can distribute over a larger part of thenetwork. There are two basic ways:


DraftJune 10, 2002


• Local distribution. This uses the local file system. The destination pro-cess has to be connected to the same file system as the source process. Thisworks well on a LAN where all machines have access to the same file system.

• Global distribution. This uses the global Web infrastructure. The filecan be put in a directory that is published by a Web server, e.g., in a~/public_html directory. The file can then be accessed by URL.

Using a URL to make connections is a general approach that works well forcollaborative applications on the Internet. To implement these techniques weneed an operation to store and load a ticket from a file or URL. This is providedby the Pickle module.

The Pickle module The Pickle module can store any value in a file andload it later. A file that has been created by Pickle is called a pickle. Whencreating a pickle, it is necessary to give a file name that is an ASCII string. Whenloading a pickle, it can be referred to either by a file name or by a URL, if thefile is stored in a place that is accessible by URL. The Pickle module has twooperations:

• {Pickle.save X FN} saves any value X in the file whose name is FN.

• {Pickle.load FNURL X} loads into X the value stored in FNURL, whichcan be a file name or a URL.

An attempt to save stateful data in a pickle will raise an exception. An attemptto save a partial value in a pickle will suspend until the partial value is complete.

Essentially, Pickle can store any information that does not change. Forexample, it can be used to store records, procedures, classes, and even functors,all of which are pure values. The following code:

{Pickle.save Coder ´ ˜/public_html/coder ´ }

saves the class Coder in a file. Files in the ˜/public_html directory are oftenpublicly-accessible by means of URLs. Anyone who needs the Coder class canjust load the file by giving the right URL:

Coder2={Pickle.load ´ http://www.info.ucl.ac.be/˜pvr/coder ´ }

Coder and Coder2 are absolutely identical from the program’s point of view.There is no way to distinguish them. Saving a value in a pickle continues to workeven if the process that did the save is no longer running. The pickle itself is afile that contains complete information about the value. This file can be copiedand transferred at will. It will continue to give the same value when loaded withPickle.load .

The main limitation of pickles is that only values can be saved. There is asimple technique for getting around this limitation that works for any languageentity, as long as the process that did the save is still running. The idea is tostore a ticket in a pickle. This works since tickets are strings, which are values.


DraftJune 10, 2002


This is a useful technique for making any language entity accessible worldwide.The URL is the entry point for the entity addressed by the ticket.

The Offer and Take operations Using Pickle and Connection , we definetwo convenience operations Offer and Take that implement this technique. Theprocedure Offer makes language entity X available through file FN:

proc {Offer X FN}{Pickle.save {Connection.offerUnlimited X} FN}

end

The function Take gets a reference to the language entity by giving the file FNURL:

fun {Take FNURL}{Connection.take {Pickle.load FNURL}}

end

This uses Pickle.load , which can load any stateless data from a file. Theargument FNURLcan either be a file name or a URL.

We can use this technique on the object C, which is an instance of the Coder

class. C can be made worldwide accessible by doing:

{Offer C ´ ˜/public_html/coderobject ´ }

Anyone who wants a reference to C need only do:

C={Take ´ http://www.info.ucl.ac.be/˜pvr/coderobject ´ }

If there is a possibility that the process that originally created the ticket is nolonger running, then the ticket has to be refreshed from a running process. Oth-erwise the ticket will no longer be valid and an attempt to connect with it willraise an exception. Section 11.6 shows one way to refresh a ticket, by using atransactional object store that has active fault tolerance.

Name servers

A general problem in open distribution is how independent computations can findeach other. The solution is to have a common infrastructure, one that is knownby both computations. Such an infrastructure is called a name server. Ticketdistribution is an instance of a name server, in which the common infrastructureis the Web. All Web servers, taken together, form a distributed, decentralizedname server, where the names are the URLs. Knowing the URL of an entitylets one find the Web server that can provide that entity. This is because thehost machine of that Web server is encoded in the URL. The global addressingprovided by the URL reposes on the IP infrastructure, which provides a uniquename for each host machine and a routing algorithm that can send informationto that machine.


DraftJune 10, 2002


Stream communication

Declarative programming with streams, as in Chapter 4, can be made distributedsimply by starting the producer and consumer in different processes. They onlyhave to share a reference to the stream.

Eager stream communication Let us first see how this works with an eagerstream. First create the consumer in one process and create a ticket for its stream:

declare Xs Sum in{Offer Xs tickfile}fun {Sum Xs A}

case Xs of X|Xr then {Sum Xr A+X} [] nil then A endend{Browse {Sum Xs 0}}

Then create the producer in another process. It takes the ticket to get a referenceto Xs and then creates the stream:

declare Xs Generate inXs={Take tickfile}fun {Generate N Limit}

if N<Limit then N|{Generate N+1 Limit} else nil endendXs={Generate 0 150000}

This creates the stream 0|1|2|3|... and binds it to Xs. This sends the streamacross the network from the producer process to the consumer process. Thisis very efficient: stream elements are sent asynchronously across the network.Because of thread scheduling, the stream is created in “batches” and each batchis sent across the network in one message.

Lazy stream communication We can run the same example with lazy streamcommunication. Take the examples with programmed triggers (Section 4.3.3) andimplicit triggers (Section 4.5) and run them in different processes, as we showedabove with eager streams. Lazy streams are not as efficient as eager streams,because messages are passed in two directions: the consumer has to ask for eachelement by sending a message.

An open server

This example uses a port to make an open server. A server is a part of anapplication that provides a service. A client of the service is any part of theapplication that uses the service. The client/server abstraction is very common;a distributed application can have many of them running simultaneously. A serveris open if a new client can connect to it at run time.

A port is a basic data type; it is a FIFO channel with an asynchronous sendoperation. Asynchronous means that the send operation completes immediately,


DraftJune 10, 2002


without waiting for the network. FIFO means that successive sends in the samethread will appear in the same order in the channel’s stream. The differencebetween ports and streams is that any number of clients can send to a port,whereas only one can “send to” a stream, i.e., add elements to the stream. It is thedifference between the declarative concurrent model and the stateful concurrentmodel.

Because of the many-to-one property, ports are ideal for making open servers.We create a port, make it globally accessible, and display the stream contentslocally:

declare S P in{NewPort S P}thread

for X in S do {Browse X} endend{Offer P serverfile}

This sets up a thread to display everything sent to the port. The for loop causesdataflow synchronization to take place on elements appearing on the stream S.Each time a new element appears, an iteration of the loop is done. Now we canlet a second process send to the port:

P={Take serverfile}thread {Send P 100} {Send P 200} endthread {Send P foo} {Send P bar} end

In the first process, 100 will appear before 200 and foo will appear before bar .The Send operation is asynchronous; it sends a message to the port withoutwaiting for a reply.

11.2.2 Network awareness

These examples show us that the distribution subsystem of Mozart is doing manydifferent things. Its implementation uses a wide variety of distributed algorithmsto provide the illusion of network transparency (e.g., see Section 11.3.4). At thispoint, you may be getting uneasy about all this activity going on behind thescenes. Can it be understood, and controlled if need be, to do exactly what wewant it to do? There are in fact two related questions:

• What are the network communications that the system uses to implementthe transparency?

• Are the network communications simple and predictable, i.e., is it possibleto build applications that communicate in predictable ways?

As we will see, the network communications are both few and predictable; inmost cases exactly what would be achieved by explicit message passing. Thisproperty of the distribution subsystem is called network awareness. Here is a


DraftJune 10, 2

002


quick summary of what happens for the most-used language entities; later on inthis chapter we will define the distributed algorithms that implement them inmore detail.

• Records, procedures, functions, classes, and functors. These are allstateless. They are copied over immediately when the ticket is taken. Thistakes one round trip, i.e., two messages. Records are copied each time theyare passed, so many copies of a record can exist on a site. On the otherhand, at most one copy of a given procedure, function, class, or functor canexist on a site.

• Dataflow variables. When binding the variable, one message is sent to thesite on which the variable was created. This site then sends one message toeach site that references the variable. This means that the system implicitlyknows all sites that reference the variable.

• Objects, cells, and locks. Objects are a particular case of distributedstate. There are many ways to implement distributed state in a networktransparent way. We single out three protocols in particular: mobile cachedobjects (the default), stationary objects, and asynchronous objects. Eachprotocol gives good network behavior for a particular pattern of use. TheCoder example we gave before defines a mobile cached object. A mobilecached object is moved to each site that uses it. This requires a maximumof three messages for each object move. Once the object has moved, nofurther messages are needed for invocations on that site. Later on, we willredo the Coder example with a stationary object (which behaves like aserver) and an asynchronous object (which allows message streaming).

• Streams. A stream is a list whose tail is an unbound variable. Sending on astream means to add elements by binding the tail. Receiving means readingthe stream’s content. Sending streams will asynchronously send streamelements from the producer process to the consumer processes. Streamelements are “batched” when possible for best network performance.

• Ports. Sending to a port is both asynchronous and FIFO. Each elementsent causes one message to be sent to the port’s home site. This kind ofsend is not possible with RMI, but it is important to have: in many cases,one can send things without having to wait for a result.

It is clear that the distributed behavior of these entities is both simple and well-defined. In a first approximation, we recommend that a developer just ignore itand assume that the system is being essentially as efficient as a human program-mer doing explicit message passing. There are no hidden inefficiencies.


DraftJune 10, 2002


proc {NewSimpleLock ?Lock}Cell = {NewCell unit }

inproc {Lock Code}Old New in

try{Exchange Cell Old New} {Wait Old} % Enter{Code} % Body

finally New=unit end % Exitend

end

Figure 11.5: Distributed locking

Distributed locking

Now that we know the distributed behavior of cells, let us it to implement a well-known distributed locking algorithm.2 When locking a critical section, multiplerequests should all correctly block and be queued, independent of whether thethreads are on the same site or on another site. We show how to implement thisconcisely and efficiently in the language. Figure 11.5, taken from Section 7.4.1,shows one way to implement a lock that handles exceptions correctly. If multiplethreads attempt to access the lock body, then only one is given access, and theothers are queued. The queue is a sequence of dataflow variables. Each threadblocks on one variable in the sequence, and will bind the next variable after it hasexecuted the lock body. Each thread desiring the lock therefore references twovariables: one to wait for the lock and one to pass the lock to the next thread.Each variable is referenced by two threads.

When the threads are on different sites, the definition of Figure 11.5 imple-ments distributed token passing, a well-known distributed locking algorithm [31].We explain how it works. When a thread tries to enter the lock body, theExchange gives it access to the previous thread’s New variable. The previousthread’s site is New’s owner. When the previous thread binds New, the ownersends the binding to the next thread’s site. This requires a single message.

11.2.3 Objects and servers

The previous section has given an example of an object and a server in a dis-tributed setting. Let us now show some more ways of using objects and servers.We treat objects and servers together since servers are closely related to objects.A common way to define a server is as a stationary object.

2The built-in locks of Mozart use this algorithm.


DraftJune 10, 2002


Stationary and mobile objects

In the Coder example, the object is mobile. This gives good performance whenthe object is shared between sites, e.g., in a collaborative application. The objectbehaves like a cache. On the other hand, it is often useful to have an object thatdoes not move, i.e., a stationary object. For example, the object might dependon some external resources that are localized to a particular site. This is a goodway to define a server, since servers are often stationary.

Whether or not an object is mobile or stationary is defined independently ofthe object’s class. It is defined when the object is created. Using New createsa mobile object and using NewStat creates a stationary object. (NewStat isdefined in Appendix B.5.) Here is a stationary version of the Coder object:

C={NewStat Coder init(100)}{Offer C tickfile}

We can see the difference in performance between the two. Do the following in asecond process, preferably on another machine:

C={Take tickfile}for I in 1..100000 do {C get(_)} end{Show done}

This does 100000 calls to C. Try doing this for both mobile and stationary objectsand measure the difference in execution time. How do you explain the difference?

Asynchronous objects and dataflow

We have seen how to share an object among several processes. Because of networktransparency, these objects are synchronous. That is, each object call waits untilthe method completes before continuing. Both mobile and stationary objects aresynchronous. For a stationary object a call requires two messages: first from thecaller to the object, and then from the object to the caller. If the network is slow,this can take long.

One way to get around the network delay is to do an asynchronous objectcall. First do the send without waiting for the result. Then wait for the resultlater when it is needed. This technique works very well together with dataflowvariables. The result of the send is an unbound variable that is automaticallybound as soon as its value is known. Because of dataflow, the variable can bepassed around, stored in data structures, used in calculations, etc., even beforeits value is known. Let us see how this works with the Coder object. First, createan asynchronous object:

C={NewAsync Coder init(100)}{Offer C tickfile}

This object has exactly the same behavior as a standard object, except that anobject call does not wait until the method completes. Assume that the secondprocess needs three random numbers and calls C three times:


DraftJune 10, 2002


Need values

Use values

Waitingtime

Call objectsynchronously when

results are needed

Need values

Use values

Call objectasynchronously when

results are needed

Need values

Use values

Call objectasynchronously before

results are needed

timeWaiting

Waitingtime

Figure 11.6: The advantages of asynchronous objects with dataflow

C1={Take tickfile}X={C1 get($)}Y={C1 get($)}Z={C1 get($)}...% use X, Y, and Z as usual

These three calls all return immediately without waiting for the network. Thethree results, X, Y, and Z, are still unbound variables. They can be used as ifthey were the results:

S=X+Y+Z{Show S}

Because of dataflow, this addition will automatically block until the three resultshave arrived. Figure 11.6 shows the three possibilities:

• In the first scenario, the program does three synchronous calls at the mo-ment it needs their results. This takes three roundtrip message delays beforeit can calculate with the results.

• The second scenario does better by using asynchronous calls. This takesslightly more than one roundtrip message delay before the calculation.

• The third scenario is best of all. Here, the asynchronous calls are initiatedbefore we need them. When we need them, their calculation is already inprogress. This can take much less than one roundtrip message delay.

The first scenario is standard sequential object-oriented programming. Becauseof dataflow, the second scenario can be done without changing any program code


DraftJune 10, 2002


except for using NewAsync instead of New to define the object. Again becauseof dataflow, the third scenario is possible without changing any program codeexcept for making the calls earlier. For both scenarios, the dataflow behavior isautomatic and can be ignored by the programmer.

Building asynchronous objects The NewAsync operation can be programmedby using New together with ports. Here is its definition:

fun {NewAsync Class Init}S P={NewPort S}Obj={New Class Init}

inthread

for M in S dotry {Obj M} catch _ then skip end

endendproc {$ M} {Send P M} end

end

Each asynchronous object has a port and a thread. A method call sends themessage to the port asynchronously. The thread reads all messages and calls theobject sequentially. If an asynchronous object call raises an exception, then theexception is lost.

Asynchronous objects with exceptions The asynchronous objects definedabove differ in one crucial aspect from standard synchronous objects. They neverraise exceptions. We can extend NewAsync slightly to handle exceptions:

fun {NewAsyncExc Class Init}S P={NewPort S}Obj={New Class Init}

inthread

for M#X in S dotry {Obj M} X= unit catch Y then X=Y end

endendproc {$ M X} {Send P M#X} end

end

Objects created by NewAsyncExc are called with two arguments, a message anda synchronization variable, e.g., like {C M X} . The variable X allows to do twokinds of synchronization:

• It is bound to unit when the object call succeeds without raising an ex-ception.

• It is bound to the exception if the object call raises an exception.


DraftJune 10, 2002


The caller can use X to check whether the execution of method Mfinishes normallyor raises an exception. Checking X can be done asynchronously with the rest ofthe program.

Separating timing from functionality

Now we have seen most of what the distributed computation model provides forobject-oriented programming. Let us summarize the possibilities. We have seenthat the same program code can be executed with synchronous objects (eithermobile or stationary) or asynchronous objects (either called on need or calledearly). In other words, the distributed model completely separates timing issuesfrom programming techniques. This is a remarkable result. It is made possiblebecause of implicit synchronization with dataflow variables.

A compute server

One of the promises of distributed computing is making computations go fasterby exploiting the parallelism inherent in networks of computers. A first step isto create a compute server, which uses its computational resources to do anycomputation that is given it. Here is one way to create a compute server:

class ComputeServermeth init skip endmeth run(P) {P} end

endC={NewStat ComputeServer init}

Assume that a client gets a reference to C, as shown before. Here is how the clientuses the compute server:

fun {Fibo N}if N<2 then 1 else {Fibo N-1}+{Fibo N-2} end

end

local F in{C run( proc {$} F={Fibo 30} end )}{Browse F}

end

local F inF={Fibo 30}{Browse F}

end

This first does the computation {Fibo 30} remotely and then repeats it locally.In the remote case, the variable F is shared between the client and server. Whenthe server binds it, its value is immediately sent to the server. This is how theclient gets a result from the server.


DraftJune 10, 2002


This compute server can execute almost any statement 〈Stmt〉 remotely. Justmake a zero-argument procedure out of it:

P=proc {$} 〈Stmt〉 end

and execute {C run(P)} . Because of network transparency, 〈Stmt〉 can be al-most any statement in the language. For example, 〈Stmt〉 can define new classesinheriting from client classes. 〈Stmt〉 can have references to other language en-tities at the client, e.g., variables for passing back results. During the remoteexecution, these references become distributed.

There is just one restriction on 〈Stmt〉: it cannot use resources. A resource isa module whose use is restricted to one site. For example, the Open module (thefile system) and the QTk module (the graphics display) are resources. In the nextsection we show how to write a compute server that can handle statements withresources.

A compute server that handles resources

The compute server shown above cannot handle statements that use resources.To execute such a statement remotely, we have to specify which resources it needs.This can be done with functors. A functor is a module specification that defineswhich resources the module needs. While functors are the basis for standalonecompilation (compiling a file that defines a functor), they are also first-class datatypes. In fact, everything that standalone compilation can do can also be donein a program. For the compute server, we can define a functor on the fly at theclient and pass it to the compute server. For example, the following statement:

declare T F infunctor F

import OSdefine

T={OS.time}end

defines a functor that is referenced by F. It specifies a module that imports theresource OSand binds T to the result of calling OS.time . The resource OScontainsbasic operating system functionality. The function OS.time returns the currenttime in seconds since Jan. 1, 1970. The functor F tells us when the computeserver thinks it is doing the computation.

To pass this functor to the compute server, we have to modify the server toaccept functors instead of procedures. That is, the server has to install the functorlocally, i.e., create a module that is connected to all the resources mentioned inthe import clause. The system module Module is able to do this. The new classComputeServer looks like this:

class ComputeServermeth init skip endmeth run(F) {Module.apply [F] _}


DraftJune 10, 2002


end

This compute server is called in exactly the same way as the previous one. Mod-

ule.apply takes a list of functors, installs them in a shared environment, andreturns a list of modules. For this application we are not interested in the result-ing module, but just in the computation.

This compute server can execute any statement 〈Stmt〉 remotely. Just makea functor out of it, specifying which resources it needs:

F=functor import 〈ResList〉 define 〈Stmt〉 end

where 〈ResList〉 is a list of resources. Then execute {C run(F)} .

Closed distribution

Closed distribution is when one application completely manages the distributionstructure. No independent application will ever connect with it. Just as thesystem module Connection provides the basic primitives for open distribution,the system module Remote provides the basic primitives for closed distribution.Remote is an extension of Module : it creates a remote site along with the meansto install new functors there. The call:

R={New Remote.manager init(fork:sh)}

creates a remote site and an object R which is the remote site’s “manager”.The argument fork:sh means the remote site is just another process on thecurrent machine. With other arguments it is possible to create processes onother machines. Calling the manager with {R apply(F X)} installs functor F

on the remote site and returns the module in X.There is a kind of “master-slave” relationship between the original site and

the new site created by Remote . The original site can observe the new site’sbehavior, for example, to keep track of its resource consumption. The originalsite can change the new site’s process priority, put it to sleep, and even terminateit if necessary. The original site can give the new site limited versions of somecritical system modules, so that the new site behaves like a sand-box.

A dynamically-upgradable compute server

Sometimes a server has to be upgraded, for example to add extra functionali-ty or to fix a bug. As a final server example, we show how to write a serverthat can be upgraded without stopping it. The upgrade can even be done in-teractively. A person sits down at a terminal anywhere in the world, starts upan interactive Mozart session, and upgrades the running server. First define ageneric upgradable server:

proc {NewUpgradableStat Class Init ?Upg ?Srv}Obj={New Class Init}C={NewCell Obj}


DraftJune 10, 2002


inSrv={MakeStat

proc {$ M} {@C M} end }Upg={MakeStat

proc {$ Class2#Init2} C:={New Class2 Init2} end }end

This definition must be executed on the server site. It returns a server Srv and astationary procedure Upg used for upgrading the server. The server is upgradablebecause it does all object calls indirectly through the cell C.

An upgradable compute server is created almost exactly as a fixed computeserver, namely by executing the following on the server site:

declare Srv Upg inSrv={NewUpgradableStat ComputeServer init Upg}

Now we can upgrade the compute server while it is running. We first define anew class CComputeServer and then we upgrade the server with an object ofthe new class:

class CComputeServer from ComputeServermeth run(P Prio<=medium)

thread{Thread.setThisPriority Prio}ComputeServer,run(P)

endend

end{Upg CComputeServer#init}

That’s all there is to it. The upgraded compute server overrides the run methodwith a new method that has a default. The new method supports the originalcall run(P) and adds a new call run(P Prio) , where Prio sets the priority ofthe thread doing the computation P.

The compute server can be upgraded indefinitely since garbage collection willremove any unused old compute server code. For example, it would be nice if theclient could find out how many active computations there are on the computeserver before deciding whether or not to do a computation there. We leave it tothe reader to upgrade the server to add a new method that returns the numberof active computations at each priority level.

Programming with distributed objects

We have seen three basic ways that an object can be distributed:

• Mobile objects, which behave like caches. Calling the object causes it to bebrought to the caller, so subsequent calls need no network operations.


DraftJune 10, 2002


• Stationary objects, which behave like synchronous servers. Calling the ob-ject requires a roundtrip message, because the caller synchronizes with theserver.

• Asynchronous objects, which are objects that receive from a port. Sendingto the port completes right away without waiting for a result. Multiplesends from the same thread arrive in the same order they were sent.

In an object-oriented program, the basic question is how does one decide whichobjects should be mobile, stationary, or asynchronous? Most objects should bestationary by default. Objects that are frequently used by remote sites can bemade mobile or asynchronous to increase performance.

11.2.4 Practical matters

Fault tolerance

Now we can write transparent distributed programs that use the network efficient-ly. But what about failure? Distributed systems often undergo partial failures.For example, the Internet suffers from very intermittent network performance:sometimes communication just freezes up. How do we handle this, while stillkeeping the good properties of network transparency? To answer this question,we first explain what kinds of failure are recognized:

• Permanent site failures that are detectable (called “permFail ”). We as-sume that when a process crashes, the crash is instantaneous and perma-nent. The crash could have many causes: a machine crash, an explicit killof the process, a software bug, or even a user manually suspending or killinga running Oz program.

• Network inactivity (called “tempFail ”). In this case, the system detectsthat communication with a given site has stopped. There’s not much morethat the Internet protocols tell us about what’s happening. We cannot evenknow whether the network will come back up again or not.

These two failure modes were chosen because they are the modes that occur mostfrequently on the Internet. Any collaborative distributed application has to takethem into account. Assume we have a distributed language entity, like an objector a port. When we use this entity, then the distribution subsystem tries to dothe operation, despite site or network failures. But if it cannot do the operation,then the subsystem should tell us the reason, i.e., whether there is a permanentsite failure or a network inactivity.

If there is network inactivity, we can always retry the operation later. Thedistribution subsystem never times out by itself; in fact the implementation’snetwork layer works hard to get around the time outs imposed by underlyingInternet protocols.


Draft

June 10, 2002


By default, a site failure or network inactivity is signaled as an exception whenone tries to do an operation. This is not always the right behavior, though. Infact, the distribution subsystem lets you do much more. We give two examplesright away: disconnected operation and failure detection. A more elaborate exam-ple, a transactional object store, is given in Section 11.6. The full failure modelis explained in Section 11.3.3. It allows you to build your own fault toleranceabstractions, e.g., the transactional store is programmed with it.

Disconnected operation Assume that you are running part of an applicationlocally on your machine through a dialup connection. These connections are notmeant to last for very long times; they are made for conversations, which usuallyare short. There are many ways a connection can be broken. For example, youmight want to hang up to make an urgent phone call, or you are connected in anairport and your calling card runs out of cash, or the phone company just dropsyour connection unexpectedly.

You would like your application to be impervious to this kind of fickleness.That is, you would like the application to wait patiently until you reconnect andthen continue working as if nothing went wrong. In Oz, you can achieve this bysetting the default failure detection to detect only permanent site failures:

% Each site executes this on startup:{Fault.defaultEnable [permFail] _}

This means that operations will only raise exceptions on permanent site failures;on network inactivity they will wait indefinitely until the network problem goesaway.

Detecting a problem and taking action On many computers, booting up isan infuriating experience. How many times have you turned on a laptop, only towait several minutes because the operating system expects a network connection,and has to time out before going on? Oz cannot fix your operating system, butit can make sure that your application will not have the same brainless behavior.

Assume you want to use a remote port. If the remote port has problems(intermittent or no access) then the application should be informed of this fact.This is easy to set up:

% Get a reference to the port:X={Take tickfile}

% Signal as soon as a problem is detected:{Fault.installWatcher X [tempFail permFail]

proc {$ _ _}{Browse X# ´ has problems! Its use is discouraged. ´ }

end }

The procedure passed to Fault.installWatcher is called a watcher; it will becalled in its own thread as soon as the system detects a problem. It’s up to you to


DraftJune 10, 2002


do what’s necessary, e.g., set an internal flag to indicate that no communicationwill be done.

If the problem was tempFail , then it’s possible that communication with X

will be restored. If that happens, Oz allows you to continue using X as if nothingwrong had happened.

Active fault tolerance Applications sometimes need active fault tolerance,i.e., part of the application is replicated on several sites and a replication algo-rithm is used to keep the parts coherent with each other. Building ADTs toprovide this is an active research topic. Because of the failure detection provid-ed by the Fault module, the Mozart system lets these ADTs be written in Ozwithout recompiling the system.

This chapter gives one example of an ADT with active fault tolerance: atransactional object store. Each instance of this ADT provides a set of objectsthat are accessed with a transactional interface. The objects are replicated onall sites that use the instance. The ADT implementation maintains coherencebetween all the replicas. Many other useful fault-tolerant ADTs can be builtwith the failure detection abilities of the Mozart system. Some of these are beinginvestigated as ongoing research in the Mozart Consortium.

Fault-detecting objects

To be useful in practice, objects must have well-defined behavior when there arefaults. There are different levels of fault tolerance that achieve different levels ofrobustness. The two functions that we have already shown, New and NewStat ,have well-defined behavior, but it may not always be what we need. They bothblock if there is network inactivity, until the network works again, and they willboth raise an exception if the object no longer exists due to a permanent sitefailure.

Persistence

As we saw above, the module Pickle lets you save any value, i.e., any statelessdata structure, in a file and load it again. The content of the file is called a pickle.If the file can be identified by a URL, then you can load it by giving this URL.Any value at all can be put in a pickle, including arbitrary data structures madeof records, literals, procedures, functors, and classes. There is one restriction:since a pickle’s content is a value, it has to be stateless, i.e., you cannot put inthings that can change. This means you cannot put in objects, ports, or dataflowvariables. This restriction is necessary because pickling is intended to be a veryfast operation. If an object changes its state then it would be quite expensive tokeep the pickle up to date. If the object is remote it would mean doing networkoperations.


DraftJune 10, 2002


To make stateful data persistent we recommend the transaction store abstrac-tion (see Section 11.6), which provides redundant fault tolerance for objects.

Debugging

Debugging is harder for distributed programs than for centralized programs. Auseful debugging technique is to use a browse server. A browse server is a dis-tributed browser, i.e., a Mozart process to which any other Mozart process cansend Browse messages. A simple browse server can be defined as follows:

functorimport Browser(browse:Browse) Distributiondefine

S P={NewPort S}thread

for X in S do {Browse X} endend{Offer P ´ ˜pvr/public_html/btick ´ }

end

Compiling and running this functor opens a browser window. To use the browseserver in a process, the process can redefine the Browse procedure locally asfollows:

localBrowsePort={Take ´ http://www.info.ucl.ac.be/˜pvr/btick ´ }

inproc {Browse X} {Send BrowsePort X} end

end

By binding the BrowsePort outside of the Browse definition, we ensure that itis loaded once only, namely when Browse is defined, and not when Browse iscalled.

Because of the asynchronous nature of ports, messages from different processesmay appear interleaved in any way in the browser window. Messages from thesame process will always appear in order, though.

Building applications

With the examples given so far in this section, you know enough already to buildfairly sophisticated distributed applications. Some things are missing, e.g., activefault tolerance with multiple processes so that one takes over if the other crashes.Later sections explain how to do this and other common tasks.

Centralized first, distributed later Developing an application is done intwo phases:


Draft

June 10, 2002


• First, write the application without partitioning the computation betweensites. Check the correctness and termination properties of the applicationon one site. Most of the debugging is done here.

• Second, place the threads on the right sites and give the objects a distributedsemantics to satisfy the geographic constraints (placement of resources, de-pendencies between sites) and the performance constraints (network band-width and latency, machine memory and speed).

The large-scale structure of an application consists of a graph of threads andobjects. Threads are created on the sites that need them. Objects may bestationary, mobile, or asynchronous. They exchange messages which may refer toobjects or other entities. Records and procedures, both stateless entities, are thebasic data structures of the application–they are passed automatically betweensites when needed. Dataflow variables and locks are used to manage concurrencyand dataflow execution.

Distributed components Functors and resources (i.e., sited modules) are thekey players in distributed component-based programming. A functor is stateless,so it can be transparently copied anywhere across the net and made persistent bypickling it on a file. A functor is linked on a site by evaluating it there with the siteresources that it needs (“plugging it in” the site). The result is a new resource,which can be used as is or linked with more functors. Functors can be used asa core technology driving an open community of developers who contribute to agrowing global pool of useful components.

Practical tips

This section gives some practical programming tips to improve the network per-formance of distributed applications: timing and memory problems, avoidingsending data that is not used at the destination and avoiding sending classeswhen sending objects across the network.

Timing and memory problems When the distribution structure of an ap-plication is changed, then one must be careful not to cause timing and memoryproblems.

When a reference X is exported from a site (i.e., put in a message and sent)and X refers directly or indirectly to unused modules then those modules will beloaded into memory. This is so even if they will never be used.

Relative timings between different parts of a program depend on the distri-bution structure. For example, unsynchronized producer/consumer threads maywork fine if both are on the same site: it suffices to give the producer thread aslightly lower priority. If the threads are on different sites, the producer may runfaster and cause a memory leak.


DraftJune 10, 2002


If the same record is sent repeatedly to a site, then a new copy of the recordwill be created there each time. This is true because records do not have globalnames. The lack of global names makes it faster to send records across thenetwork.

Avoid sending useless data When sending a procedure over the network,be sure that it does not contain calculations that could have been done on theoriginal site. For example, the following code sends the procedure P to remoteobject D:

R={MakeTuple big 100000} % A very, very big tupleproc {P ?X} X=R.2710 end % Procedure that uses field 2710{D addentry(P)} % Send P to be executed

If D executes P, then the big tuple R is transferred to D’s site, where field number2710 is extracted. With 100,000 fields, this means 400KB is sent over the network!Much better is to extract the field before sending P:

R={MakeTuple big 100000}F=R.2710 % Extract field 2710 before sendingproc {P ?X} X=F end % Procedure that uses field 2710{D addentry(P)}

This avoids sending the tuple across the network. This technique is a kind ofpartial evaluation. It is useful for almost any Oz entity, for example procedures,functions, classes, and functors.

Avoid sending classes When sending an object across the network, it is goodto make sure that the object’s class exists at the destination site. This avoidssending the class code across the network. How can we do this in the case of acollaborative tool? Two sites have identical binaries of this tool, which they arerunning. The two sites send objects back and forth. Here is how to write theapplication as a functor:

class C% ... lots of class code comes here

endfunctordefine

Obj={New C init}% ... code for the collaborative tool

end

This creates the class C first and then references it inside the functor. This meansthat all copies of the binary with this functor will reference the same class, sothat an object arriving at a site will recognize the same class as its class on theoriginal site.

Here is how not to write the application:


DraftJune 10, 2

002

11.3 The distribution model (part II) 663

functordefine

class C% ... lots of class code comes here

endObj={New C init}% ... code for the collaborative tool

end

Do you see why? Think first before reading the next paragraph!In both solutions, the functor is applied when the application starts up. In

the second solution, this defines a new and different class C on each site. If anobject of class C is passed to a site, then the site will ask for the class code to bepassed too. This can be very slow if the class is big–for a full-featured applicationit can make a difference of several minutes on a typical Internet connection. Forexample, the functionality of the TransDraw editor in the Mozart release takesabout 1 MB of compiled code, which all has to be sent. In the first solution, classC is defined before the functor is applied. When the functor is applied, the classalready exists! This means that all sites have exactly the same class, which ispart of the binary on each site. Objects passed between the sites will never causeclass code to be sent.

11.3 The distribution model (part II)

The previous section has shown the basics of programming with the distributedcomputation model. Now it is time to study the model in more depth. Thegoal of the model is to support a network-transparent system with simple andpredictable network communications. To achieve this, we have implemented eachlanguage entity with a distributed algorithm designed especially for it.

As a starting point, we take advantage of the distinction between stateless,single-assignment, and stateful entities. This distinction already exists in thelanguage to classify different computation models, as shown in Chapters 2 to 7.It was a serendipitous discovery that this distinction is just as important fordistribution. This is because state and distribution do not mix well. State changeshave to be visible globally, which is expensive to implement in a distributedsystem. Having weaker forms of state, namely the stateless and single-assignmententities, is therefore important for efficiency in a distributed system.

Historical background

The work on transparent distribution in Oz started in the PERDIO project atSaarbrucken in early 1995. The project proposal stated hopefully [149]:

Having both distribution and persistence in a single system gives syn-ergy. The multitude of separate models (programming-language mod-el, long-term storage model, distribution model) is replaced by a single


DraftJune 10, 2002


model. Where previously explicit code had to be written to move databetween models, in the single model none is needed. Where previous-ly augmenting a program with the abilities of another model requiredmajor effort (e.g., making a program distributed), in the single mod-el, a program can be written with only the single model in mind,and with minor changes it will have the abilities of the other mod-els. The combination of persistence and distribution is of particularimportance for the construction of fault-tolerant software systems.

It was not clear that these goals could be achieved. Nontrivial distributed al-gorithms had to be devised, and even worse, it was not at all obvious that thebasic goal (separating the different models) was practical. The first design thatdefined all the distributed algorithms needed for the stateful concurrent modelwas completed in 1996 [41]. The first public release of the Mozart system, whichimplements an extended version of this model, occurred in January 1999 [110]. Toa large degree, this system realizes the goals set forth in the PERDIO proposal.

Facts of life in distribution

There are several “facts of life” that it is important to respect in distributedprogramming.

• Avoid global operations, e.g., no global synchronization. Global operationsare extremely expensive, especially if partial failure is possible. The dis-tributed algorithms used to implement the language operations should be“local”, in the sense that they require the minimum number of sites.

This rule is hard to follow for distributed garbage collection: to our knowl-edge there exists no algorithm that can collect all garbage eventually andthat does not require global synchronization. As a result, it is the program-mer’s responibility to remove cross-site cycles between stateful entities. Thisturns out to be reasonably easy to do, e.g., a client should remove any ref-erences to the server when it no longer needs the server, or vice versa. Ifthe server is correctly written (it removes client references) then the clientneeds no particular precaution.

• Avoid third-party dependence (Lamport’s rule). It should never be thecase that the failure of an uninvolved site cause problems between sitesinvolved in a distributed computation. This rule was made popular byLeslie Lamport.

• Support latency tolerance (network fluctuations). Because network perfor-mance is unpredictable, the application should never rely on a particularminimum level of performance. It should be “downward scalable” in thesense of reacting appropriately to arbitrarily small network bandwidth andarbitrarily large network latency. (See section on time outs, below.)


DraftJune 10, 2002


• Support temporary network inactivity. There are problems with precisefailure detection – sometimes, the network becomes inactive with a lack ofinformation as to why. The system should be able to handle this case. TheInternet uses TCP on a WAN (wide-area network) and therefore does nothave precise failure detection. TCP on a LAN (local-area network) doeshave it. This makes fault tolerance very different on a LAN from on aWAN.

• Be aware that causality, execution order, and consistency are different in adistributed system with respect to a centralized system.

Time outs considered harmful

Many existing systems (both programming platforms and applications) do nothandle time correctly. When there is a prolonged network inactivity during anoperation, they do a time out: they abort the waiting operation and invokean error handling routine. Often, this abort is irrevocable: it is impossible tocontinue the operation. Many operating system utilities are of this type, e.g.,ftp. The mistake in this approach is that the decision to time out is made at thewrong level. For example, assume there is a time out in a lower layer, e.g., thetransport layer (TCP protocol) of the network interface. This time out crossesall abstraction boundaries to appear at the top level, i.e., to the user. The user isinformed in some way: the application stops, or at best a window is opened askingconfirmation to abort the application. The user does not have the possibility tocommunicate back to the timed-out layer. This greatly limits the flexibility ofthe system.

The right approach is not to time out by default but to let the applicationdecide. The application might decide to wait indefinitely, avoiding an abort, or toabort immediately without waiting. In many cases, neither of these possibilitiesis offered. Whenever they are offered, the perceived quality of the system ismuch improved. A hard-mounted resource in the NFS file system offers the firstpossibility. The Stop button in the Netscape Web browser offers the secondpossibility.

A simple way to implement this is for the system to signal the applicationwhen there is a network inactivity, without aborting any pending operations.The signal is given concurrently, in its own thread. The application then is freeto decide what to do. An major advantage of this approach is that the applicationcan react much quicker to a network inactivity. This is because the signal is non-destructive. Because the signal is benign, the time delay for the signal does nothave to be an approximation to infinity, like the duration of a time out. Thesignal can be given quickly, as soon as a lull in network activity is detected.


DraftJune 10, 2

002


11.3.1 Global naming

An important issue in distributed computing is naming. How do independentcomputations avoid confusion when communicating with each other? They do soby using globally-unique names for things. For example, instead of using printrepresentations (character strings) to name classes, globally-unique names is usedinstead. The uniqueness should be guaranteed by the system. We distinguish fourkinds of name in the distributed computation model:

• References. A reference is an unforgeable means to access any languageentity. To programs, a reference is transparent, i.e., it is dereferenced whenneeded to access the entity. References can be local, to an entity on thecurrent site, or remote, to an entity on a remote site. For example, a threadcan reference an entity that is localized on another site. The language doesnot distinguish local from remote references.

• Tickets. A ticket, in Mozart terminology, is a global means to access anylanguage entity. A ticket is similar to a reference, except that it is validfor any Mozart process, it is represented by an ASCII string, it is explic-itly created and dereferenced, and it is forgeable. A computation can geta reference to an independent computation by getting a ticket from thatcomputation. The ticket is communicated using any communication proto-col between the processes (e.g., TCP, IP, SMTP, etc.) or between the usersof these processes (e.g., sneakernet, telephone, PostIt notes, etc.). Usually,these protocols can only pass simple datatypes, not arbitrary language ref-erences. But in almost all cases they support passing information coded inASCII form. If they do, then they can pass a ticket.

• Names. A name is an unforgeable constant that is used to implementabstract data types. Names can be used for different kinds of identity andauthentication abilities (see Sections 3.7.4 and 5.4). All language entitieswith name equality, e.g., objects, classes, procedures, functors, etc. (seeChapter 13), implement their identity by means of a name embedded insidethem.

• URLs (Uniform Resource Locators). A URL is a global reference toa file. The file must be accessible by a World-Wide Web server. A URLencodes the hostname of a machine that has a Web server and a file name onthat machine. URLs are used to exchange persistent information betweenprocesses. A common technique is to store a ticket in a file addressed byURL.

Within a distributed computation, all these four kinds of names can be passedbetween sites. References and names are pure names, i.e., they do not explicitlyencode any information other than being unique. They can be used only inside adistributed computation. Tickets and URLs are impure names since they explic-itly encode the information needed to dereference them–they are ASCII strings


DraftJune 10, 2

002


and can be read as such. Since they are encoded in ASCII, they can be used bothinside and outside a distributed computation.

11.3.2 Programmer support

Essentially all the distribution abilities of Oz are given by the following fourmodules:

• The module Connection provides the basic mechanism for translating lan-guage references into tickets and vice versa. This makes it possible forindependently-running applications to connect with each other.

• The module Remote allows an active application to create a new site (i.e.,a local or remote operating system process) and install functors on it. Thenew site may be on the same machine or a remote machine. The newsite is the basic unit for encapsulating computations for reasons of securityor resource management. If the new site is local, then it shares memorypages with the original site, which makes site communication an order ofmagnitude faster than ordinary interprocess communication.

• The module Pickle allows an application to store stateless data in files andretrieve it from both files and URLs. Retrieval can be done with either a filename or a URL. Any stateless data can be saved, including records, names,procedures, functions, functors, and classes. This makes it unnecessary forthe application to do any coding or decoding of data.

• The module Fault gives the basic primitives for failure detection and han-dling. It allows both synchronous and asynchronous detection of both per-manent and temporary failures of language entities. This allows to buildfault tolerance abstractions completely in the model. This book describesa few useful abstractions, e.g., in Sections 11.5 and 11.6. The developmentof more powerful ones is still ongoing research.

There is a fifth module that has nothing to do with distribution specifically, butthat is very useful when used with functor values:

• The module Module provides the basic mechanism to install functors. Afunctor specifies a module along with the resources it needs. Installing afunctor will execute the functor body, which can initialize the module anddo any other required computation. Applications are built out of functors.A resource is a module that can be used only on a particular site. Functorsare first-class values; they can be created dynamically, stored in files, passedin arguments, and dynamically installed on any site.

The four modules Connection , Module , Remote , and Pickle are extremely easyto use. In each case, there are just a few basic operations. Connection has two:offering a ticket and taking a ticket, Pickle has two: saving and loading a pickle,


DraftJune 10, 2

002


and Module and Remote both have one: installing a functor. Because of networktransparency, there is no need to change any program code when changing aprogram’s distribution structure with Connection , Module , or Remote .

11.3.3 Failure model

Distributed systems have the partial failure property, that is, part of the systemcan fail while the rest continues to work. Partial failures are not at all rare.Properly-designed applications must take them into account. This is both goodand bad for application design. The bad part is that it makes applications morecomplex. The good part is that applications can take advantage of the redundan-cy offered by distributed systems to become more robust. One of the challengesof distributed programming is to minimize the extra complexity while making iteasy to take advantage of the redundancy.

The first step in programming with partial failure is how the failure is seenby the program. The failure model defines what failures are recognized by thesystem and how they are reflected in the language. We have designed a simplebut useful failure model. The system can detect two kinds of failures, permanentsite failure and network inactivity:

• Permanent site failure is more generally known as fail-silent with failure de-tection. It is indicated by the failure mode permFail . That is, a site stopsworking instantaneously, does not communicate with other sites from thatpoint onwards, and the stop can be detected from the outside. Permanentsite failure cannot in general be detected on a WAN (e.g., the Internet), butonly on a LAN.

• Network inactivity is a kind of temporary failure. It can be either temporaryor permanent, but even when it is supposedly permanent, one could imaginethe network being repared. This is different from site failure because thenetwork does not store any state. Network inactivity is indicated by thefailure mode tempFail . The Mozart system assumes that it is alwayspotentially temporary, i.e., it never times out by default.

These failures are reflected in the language in two ways, either synchronously orasynchronously:

• Synchronous (i.e., lazy) detection is done when attempting to do an oper-ation on a language entity. If the entity is affected by a failure, then theoperation is replaced by another, which is predefined by the program.

• Asynchronous (i.e., eager) detection is done independent of any operations.A program first posts a “watcher” on an entity, before any problems occur.Later, if the system detects a problem, then it enables the watcher, whichexecutes a predefined operation in a new thread.

The two failure modes, taken together with the two detection mechanisms, aresufficient for writing fault-tolerant distributed applications.


DraftJune 10, 2002


Network inactivity

The network inactivity failure mode allows the application to react quickly totemporary network problems. It is raised by the system as soon as a networkproblem is recognized. It is therefore fundamentally different from a time out.For example, TCP gives a time out after some minutes. This duration has beenchosen to be very long, approximating infinity from the viewpoint of the networkconnection. After the time out, one can be sure that the connection is no longerworking.

The purpose of tempFail is to inform the application of network problems,not to mark the end of a connection. For example, if application is connectedto a server and if there are problems with the server, then the application wouldlike to be informed quickly so that it can try connecting to another server. AtempFail failure mode can therefore be relatively frequent, much more frequentthan a time out. In most cases, a tempFail fault state will eventually go away.

It is possible for a tempFail state to last forever. For example, if a userdisconnects the network connection of a laptop machine, then only he or sheknows whether the problem is permanent. The application cannot in generalknow this. The decision whether to continue waiting or to stop the wait can cutthrough all levels of abstraction to appear at the top level (i.e., the user). Theapplication might then pop up a window to ask the user whether to continuewaiting or not. The important thing is that the network layer does not make thisdecision; the application is completely free to decide or to let the user decide.

11.3.4 Implementation

In this section we briefly summarize how the distribution model is implemented inMozart. This implementation extends the centralized language implementationin two ways. Each language entity has one or more distributed algorithms toimplement its distributed behavior. There is also a distributed garbage collector.We explain each in turn.

Language entities

Each language entity is implemented by one or more distributed algorithms. Eachalgorithm respects the entity’s semantics if distribution is disregarded. We firstsummarize all the protocols briefly. Afterwards we will then focus on two of themore interesting protocols, for mobile state and distributed binding, to explainthem in more detail. The language entities have the following protocols:

• The stateful entities are implemented with one of the following three pro-tocols:

– Stationary state. All operations always execute on the same site.Remote invocations send messages to this site. Both asynchronousand synchronous sends are possible.


DraftJune 10, 2

002


– Mobile state. The right to update the state can change sites. Wecall this right the state pointer or content edge. The basic mobilestate protocol behaves like a cache, where all operations execute inthe thread that invokes them. An exchange operation will first causethe state pointer to move to the executing site, so that the exchangeis always local [167, 164].

– Invalidation. The right to update the state can change sites. A stateupdate is immediately broadcast to all sites referencing the entity.

• The single-assignment entities are implemented with a distributed unifica-tion algorithm [65]. The key operation of this algorithm is a distributedbind operation, which replaces the variable by whatever it is bound to, onall the sites that know the variable. There are two variants of this algorithm:

– Lazy binding (on demand). The replacement is done on a site onlywhen the site needs the variable. This variant reduces the number ofnetwork messages, but increases latency and keeps a dependency onthe variable’s home site.

– Eager binding (on supply). The replacement is done on a site as soonas the variable is bound, whether or not the site needs the variable.

The Mozart system currently implements just the eager binding algorithm.

• The stateless entities are implemented with one of the following three pro-tocols [4]:

– Lazy copying (on demand). The value is copied to a site only whenthe site needs it. This reduces the number of network messages, butincreases latency and keeps a dependency on the original site.

– Eager copying (on supply, sent if not present). The value is notsent as part of the message, but if upon reception of the message, thevalue is not present, then an immediate request is made for it. In mostcases, this is the optimal protocol, since the value will be present onthe receiving site for all except the first message referencing it.

– Eager immediate copying (on supply, always sent). The value issent as part of the message. This has minimum latency, but can over-load the network, since values will be repeatedly sent to sites. Thisprotocol is only useful for doing initializations.

• In addition to these algorithms, there is a distributed garbage collectionalgorithm. This algorithm works alongside the local garbage collection.The algorithm does no global operations and is able to remove all garbageexcept for cross-site cycles between stateful entities. The algorithm uses acredit mechanism, which is kind of weighted reference counting [121]. Eachlanguage entity with remote references has a supply of “credits”. Each


DraftJune 10, 2002


Kind of entity Algorithm EntityStateless Eager immediate copying record, integer

Eager copying procedure, class, functorLazy copying object-record

Single assignment Eager binding dataflow variable, streamStateful Stationary state port, thread, object-state

Mobile state cell, object-state

Table 11.1: Distributed algorithms

remote reference to the entity must have at least one credit. When localgarbage collection removes the remote reference, then its credits are sentback. When the entity has no outstanding remote credits and no localreferences, then it can be reclaimed.

Table 11.1 shows the algorithms used by the current system for each languageentity. For each of these entities, network operations3 are predictable, which givesthe programmer the ability to manage network communications.

Mobile state protocol

The mobile state protocol is one of the distributed algorithms used to implementstateful entities. Objects, cells, and locks are implemented with this protocol.This section gives the intuition underlying the protocol and explains the networkoperations it does. A formal definition of the protocol and a proof that it respectsthe language semantics are given in [167]. An extension that is well-behaved incase of network and site failure is given together with proof in [18]. The Mozartsystem implements this extended protocol.

We use a graph notation to describe the protocol. Each (centralized) languageentity, i.e., record, procedure value, dataflow variable, thread, and cell, is repre-sented by a node in the graph. To represent a distributed computation, we addtwo additional nodes, called proxy and manager. Each language entity that hasremote references is represented by a star-like structure, with one manager and aset of proxies. The proxy is the local reference of a remote entity. The managercoordinates the protocol that implements the distribution behavior of the entity.

Figure 11.7 shows a cell that has remote references on three sites. The cellconsists of three proxies P and one manager M. The cell content X is accessiblefrom the first proxy through the state pointer. A thread T on the third sitereferences the cell, which means that it references the third proxy.

What happens when T does an exchange operation? The state pointer isone a different site from T, so the mobile state protocol is initiated to bring thestate pointer to T’s site. Once the state pointer is local to T, then the exchange

3In terms of the number of network hops.


DraftJune 10, 2002


state pointer

cell content

T

M

P

X

PP

Figure 11.7: Graph notation for a distributed cell

state pointer

cell content

T

M

P

X

PP

get(1)forward

put(3)

(2)

Figure 11.8: Moving the state pointer

is performed. This implies the remarkable property that all cell operations arealways performed locally in the thread that initiates them.

The protocol to move the state pointer consists of three messages: get, put,and forward. Figure 11.8 shows how they are sent. The third proxy initiates themove by sending a get request to M. The manager M plays the role of a serializer:all requests pass through it. After receiving the get, M sends a forward messageto the first proxy. When the first proxy receives the forward, it sends a put tothe third proxy. This atomically transfers the state pointer from the first to thethird proxy.

Distributed binding protocol

The distributed binding protocol is used to bind dataflow variables that havereferences on several sites. The general binding algorithm is called unification;the distributed version does distributed unification. A formal definition of theprotocol and a proof that it respects the language semantics are given in [65].The Mozart system implements an extended version of this protocol that is well-behaved in case of network and site failure.

When unification is made distributed it turns out that the whole algorithmremains centralized except for one operation, namely binding a variable. To givethe intuition underlying distributed unification it is therefore sufficient to explaindistributed binding.

Figure 11.9 shows a dataflow variable V that has remote references on three


DraftJune 10, 2

002


1 2T T

M

P

PP

Figure 11.9: Graph notation for a distributed dataflow variable

1 2T T

P PM

P

requestbinding

binding(2)(2)

binding

(1)(2)

W=V+1 V=10

Figure 11.10: Binding a distributed dataflow variable

sites. Like the distributed cell, there are three proxies P and one manager M.The manager has references to all proxies. On the first site, thread T1 referencesV and is suspended on the operation W=V+1. On the third site, thread T2 alsoreferences V and is about to do the binding V=10.

The protocol to bind V consists of two messages: request(X) and binding(X).Figure 11.10 shows how they are sent. The third proxy initiates the protocolby sending request(10) to M. The first such request received by M causes abinding(10) to be sent to all proxies. This action is the heart of the algorithm.The rest is details to make it work in all cases. If M is already bound when itreceives the request, then it simply ignores the request. This is correct since theproxy that sent the request will receive a binding in due course. If a new proxyis created on a fourth site, then it must register itself with the manager. Thereare a few more such cases; they are all explained in [65].

This algorithm has several good properties. In the common case where thevariable is on just two sites, for example where the binding is used to return theresult of a computation, the algorithm’s latency is a single round trip. This isthe same as explicit message passing. A binding conflict (an attempt to bind thesame variable to two incompatible values) will cause an exception to be raised onthe site that is responsible for the conflict.


DraftJune 10, 2002


Memory management

When distribution is taken into account, the Mozart system has three levels ofgarbage collection:

• A local garbage collector per site. This collector coordinates its work withthe distributed collectors.

• A distributed garbage collector that uses weighted reference counting [121].

• A distributed garbage collector based on a time-lease mechanism.

Weighted reference counting The first level of distributed garbage collectionuses weighted reference counting [121]. This collector works when there are nofailures. It can rapidly remove all distributed garbage except for cross-site cyclesbetween stateful entities on different owner sites. Each remote reference has anonzero amount of credit, that implies the continued aliveness of the entity. Whenthe remote reference is reclaimed, the credit is returned to the owner. When theowner sees there is no longer any outstanding credit, then the entity can bereclaimed if there are no more local references.

Weighted reference counting is efficient and scalable. First, creating a newremote reference requires essentially zero network messages in addition to themessages sent by the application. Second, each remote site does not need toknow any other site except the owner site. Third, the owner site does not needto know any remote site.

Time-lease mechanism The second level of distributed garbage collectionuses a time-lease mechanism. This collector works when there are permanent orlong-lived temporary failures. Each remote reference has only a limited lifetime(a “lease on life”), and must periodically send a “lease renewal” message to theowner site. If the owner site does not receive any lease renewal messages after agiven (long) time period, then it assumes that the reference may be considereddead.

The time-lease mechanism is complementary to weighted reference counting.The latter reclaims garbage rapidly in the case when there are no failures. Theformer is much slower, but it is guaranteed in finite time to reclaim all garbagecreated because of failure. This plugs the memory leaks due to failure.

Programmer responsibility The main problem with distributed garbage col-lection is to collect cycles of data that exist on several sites and that containstateful entities. As far as we know, there does not exist an efficient and simpledistributed garbage collector that can collect these cycles.

This means that distributed memory management is not completely auto-matic; the application has to do a little bit of work. For example, consider aclient/server application. Each client has a reference to the server. Often, the


DraftJune 10, 2002

11.4 Some related concepts 675

server has references to the clients. This creates a cycle between each client andthe server. If the client and server are on different sites, they cannot be reclaimed.To reclaim a client or a server, it is necessary to break the cycle. This has to bedone by the application. There are two ways to do it: either the server has toremove the client reference or the client has to remove the server reference.

Creating a ticket with {Connection.offerUnlimited X T} makes X a per-manent reference. That is, X is added to the root set and is never reclaimed.Once-only tickets, i.e., those created with {Connection.offer X T} , can onlybe taken once. As soon as they are taken, X is no longer a permanent referenceand is potentially reclaimable again.

It is possible to use distribution to reduce the time needed by local garbagecollection. With Remote , create a small process that runs the time-critical partof the application. Since the process is small, local garbage collection in it willbe very fast.

11.4 Some related concepts

11.4.1 Parallelism

We say a program executes in parallel if it is run on hardware that permitsmore than one operation to be executed simultaneously. If execution time is toohigh, then parallel execution is one way to speed things up. This can be doneon many levels, e.g., the processor can execute multiple instructions in parallelor the program can be partitioned among multiple processors. Restructuring aprogram to get good parallel performance can be quite difficult. If executiontime is satisfactory, then we do not recommend doing parallel execution. For thissection, assume that we do need higher performance and that we want to do theeffort to get parallel execution.

One of the simplest ways to get parallel execution is to use a set of computersconnected by a network. Networked computers are ubiquitous nowadays andoften of very high performance. Current “supercomputers”, defined as the fastestexisting computers at any given time, are in fact clusters: fast computers linked byfast point-to-point networks. Clusters are important for problems that need thehighest performance. But for many problems, standard technology is sufficient,i.e., standard workstations connected by a standard network.

A distributed execution is a parallel execution if speedup is possible, i.e., thesites execute on different processors. Parallel execution implies that the program’sexecution is partitioned over the sites. An important factor in the resultingparallel performance is the communications overhead. When one site sends datato another site, then the data’s internal representation has to be translated intoa byte stream, which is sent over the network, and which at the destination istranslated back to the internal representation. This can give a very big overhead,so much so that only very “coarse-grained” parallelism is possible, i.e., parallelexecution of rather big parts of the program. Because the distribution model is


DraftJune 10, 2002


network transparent, the translation is not seen by the programmer. Only thereduced performance gives it away.

Because of network transparency, it is easy to partition the program overmultiple sites. The hard part is deciding which part to execute where so thatcommunications overhead is low while each processor is still kept busy most ofthe time.

11.4.2 Mobility

The term “mobility” covers many different concepts. First of all, there is thedistinction between mobile computing and mobile computations:

• Mobile computing means that the computer itself moves and changes itsattachment point to the network, during the execution of applications. Thisis not support by standard Internet protocols, such as IPv4. It requiresprotocol extensions such as Mobile IP.

• Nomadic computing is a restricted form of mobile computing, in whichthe applications have to be stopped and restarted whenever the comput-er changes its attachment point. This is supported by standard Internetprotocols. A computer can dynamically obtain a new IP address when itreconnects to a network.

• Mobile computation means that a computation moves from one computer toanother. The difference between mobile computing and mobile computationis that in the first case the computer moves and in the second case thecomputer does not move.

In this book, we will be concerned only with mobile computations. Even withthis restriction, the term mobility covers many different kinds of mobility. Hereis a partial classification, from “weakest” to “strongest”:

• Mobile code means that the compiled instructions of a program fragment(class, package, procedure, etc.) are transferred between processes. Noexecution state is transferred.

• Mobile closures means that a procedure value is transferred between pro-cesses. A procedure value can reference execution state.

• Weak mobility means that the programmer-visible state of a computationis atomically transferred at explicit points during the computation. Mobileprocedure values are a form of weak mobility.

• Strong mobility means that all the execution state of a computation (in-cluding thread stacks) is atomically transferred at explicit points duringthe computation.


DraftJune 10, 2002

11.5 Generic client/server 677

An important issue in mobile computation is resource management. Each sitehas different resources, which can only be used on that site. A computationthat moves needs to unplug the resources of the source site and plug in theresources of the destination site. Furthermore, this should be done atomically.The complexity of this operation means that strong mobility is not as simple asit appears on first glance. In practice, it turns out that strong mobility is rarelyneeded in its complete form.

Mobile objects

One of the important parts of the distribution model is the ability to tune thecommunication overhead of a language entity independent of the entity’s languagesemantics. In particular, for objects, the model can cache object state. The objectstate can move. To be precise, the object’s state pointer, defined as the right tochange the object state, will move to the site that is using the object. If the statepointer is on your site, then an exchange is a purely local operation. If the statepointer is on another site, then the state pointer is moved to your site beforedoing the exchange.

Mobile objects are useful for performance optimization, not for writing mobileapplications. For the latter, see the next section.

Mobile agents

The simplest way to do mobile agents in Oz is to create new functors on the flyand install them on new sites. An executing agent does not actually move, butit can create an extension of itself on another site. The management of resourcesis done explicitly.

There are several ways to make this approach fault tolerant. The simplestway is to store the agent state in a pickle. A more sophisticated way is to storethe agent state in a fault-tolerant medium, for example, in a transactional store(see Section 11.6).

11.5 Generic client/server

A server is a long-lived computation that provides a service to clients. We havealready shown how to build a simple servers and make them accessible to clients.This section presents a generic client/server tool that takes any centralized, se-quential client/server application written in Oz and, without writing any newcode, turns it into an open, robust, distributed, concurrent client/server appli-cation. Any number of clients can connect dynamically to the server and askquestions concurrently. The tool guarantees that all client requests are serializedwhen fed to the server.

The server tolerates client failures (it silently drops failed clients) and clientscan reconnect to a new server if the server fails (a graphic user interface is pro-


DraftJune 10, 2002


Enhanced application (open, robust, distributed, concurrent):

Centralized, sequential client-server application:

The application must consist of twoindependent parts: a front end (Client)that queries a back end (Server)

●

Server

URL

dserver

dclient makes Client into a reusable client

dserver makes Server into an open server●

●

Client Server

Client dclient

Client dclient

according to a well-defined interface

Needs only compiled Client & Server●

Figure 11.11: Generic client/server

vided to handle the reconnection process). This is adequate for client/serverapplications without server state.

11.5.1 Application interface

To use the tool, the application must be logically divisible into two parts: a serverand a client (see Figure 11.11). Typically, the client has the user interface andthe server has the engine that does the calculation. The interface between thetwo must be through an object or a one-argument procedure, {S M} , which whengiven passed a message M, will bind any variables in M to return the result. Itdoes not matter what the types of the argument and result are, except that theymust be stateless. That is, they can be any combination of data structures thatdo not change (like records, lists, numbers, atoms, strings, procedures, functions,functors, and classes) but they should not contain any stateful data (like objects,ports, dictionaries, and resources).

Format of client and server

The server and the client must be written as functors, e.g., Server.oz andClient.oz, that respect the correct interfaces. Server.oz must respect the fol-lowing interface:

functorimport ...


DraftJune 10, 2

002


exportserver:S

define...

end

S is an object or one-argument procedure that provides the stateless server func-tionality. It must be defined in the functor body. Client.oz must respect thefollowing interface:

functorimport ...export

server:Sreconnect:R

define...

end

S is a one-argument procedure that is defined by the client/server tool and thatcan be called in the functor body. R is an optional zero-argument procedure thatis called internally by the client/server tool when the client connects to anotherserver.

Using the tool

The client/server tool consists of two programs, dserver and dclient. Theseprograms are implemented with the Fault module (see Appendix B.5.2 for thecomplete implementation). To use the tool, first compile the client and server,giving Client.ozf and Server.ozf. Then to start a server, call dserver as follows:

% dserver --in=Server.ozf --publish=filename &

This takes the compiled functor Server.ozf and a file name as input. It starts aserver and publishes an entry point in the file. The server must have write accessto the file. Typically, the file will be accessible world-wide through an URL. Tostart a client, call dclient as follows:

% dclient --in=Client.ozf --connect=FNURL &

This takes the compiled functor Client.ozf and a file name or URL as input.It finds the server at the given file name or URL (we suppose that it correspondsto the file used by the server), and starts up a client. The client must have readaccess to FNURL.

11.5.2 A distributed expression evaluator

We give a simple example application to show how the client/server tool works.The application is an expression evaluator: the client types expressions which are


DraftJune 10, 2002


Figure 11.12: A distributed expression evaluator

evaluated by the server and the results are displayed by the client. Figure 11.12gives a screenshot of the client interface. The functor Server.oz is defined asfollows:

functorimport Compilerexport server:Sdefine

proc {S M}try

M.2={Compiler.evalExpression M.1 env _}catch X then

raise {AdjoinAt X debug removed} endend

endend

This uses the Compiler module to compile and evaluate the expression. Anexception X is raised if the expression has a syntax error. We use AdjoinAt toreplace X’s debug field by another debug field that contains only the atom re-

moved. This is because the debug field contains a huge amount of information(the thread’s stack, which indirectly points to a large number of library opera-tions), which would otherwise be transferred across the network, causing a largetransmission delay. This is one of those details that is important for practicalnetwork-transparent distribution. The functor Client.oz is defined as follows:

functorimport Application QTkexport server:Sdefine

In Out H


DraftJune 10, 2002


D=td(title:"Super expression evaluator"tdrubberframe(glue:nswe

lr(glue:nswelabel(text:"Expression")text(handle:In glue:nswe))

lr(glue:nswetd(label(text:"Result")

label(text:"DONE" bg:green handle:H))text(handle:Out glue:nswe tdscrollbar: true )))

lr(glue:webutton(glue:we text:"Eval"

action: proc {$}try E V in

{H set(text:"CALCULATING" bg:red)}{In get(E)} {S E#V} {Out set(V)}

catch _ then{Out set("*** Syntax Error ***")}

finally {H set(text:"DONE" bg:green)} endend )

button(glue:we text:"Quit"action:toplevel#close)))

W={QTk.build D}thread {W show(wait: true )} {Application.exit 0} end

end

The client is just a graphic front end to the server. The client gets the expressionby calling {In get(E)} , then calls V={S E} to evaluate it, and finally displaysthe result with {Out set(V)} . All the real work is done by S. If there is anexception, then it displays a message saying there was a syntax error.

Despite its apparent simplicity, this example application is quite powerful. Ithas the full power of Mozart’s incremental compiler and arbitrary precision integerarithmetic. For example, you can calculate all digits of 100! (100 factorial) byentering the following expression:

localfun {F N} if N==0 then 1 else N*{F N-1} end end

in {F 100} end

11.5.3 A chat room

We now show how to program a simple but useful chat room application usingthe client/server tool. Figure 11.13 shows the architecture of the chat room withits objects and the messages sent between them. The figure gives two scenarois: aclient adding a line of text and a new client joining the chat room. The server S isan instance of the class ChatServer , which has three methods (see Figure 11.14):

• {S init} initializes the server (empty conversation and no clients).


DraftJune 10, 2002


C4

C3

C2 S

Chat server object

Chat client objects

3 and 4: a new client registers with theserver, which then sends it the current text

●

1 and 2: a new line of text is sent to theserver, which broadcasts it to all clients

●

1. newline(L)

2. newline(L)

Message send

Active object4. T=<current text>

3. newclient(C4 T)

C1

Figure 11.13: Architecture of the chat room application

• {S newclient(C T)} registers the client, i.e., it adds client C to the serv-er’s list of clients and it binds T to the current conversation (all previoustext).

• {S newline(L)} adds the text line L to the conversation stored by theserver.

Each client C is an instance of the class ChatClient , which has three methods(see Figure 11.15):

• {C init(N S C)} initializes a client with its name N and server S. Thiscreates the client window where the client can see the conversation and cansend lines of text. This also registers the client with the server and displaysthe current conversation.

• {C newline(L)} adds the text line L to the conversation displayed by theclient.

Again we use the QTk module of Chapter 10 to create the graphic user interface.The server is created as follows:

S={NewActive ChatServer init}

Clients can then be created as follows:

C1={New ChatClient init("Peter" S C1)}C2={New ChatClient init("Seif" S C2)}


DraftJune 10, 2002


class ChatServerattr

textlinesclients

meth inittextlines:=nilclients:=nil

end

meth newclient(C T)clients:=C|@clientsT={Reverse @textlines}

end

meth newline(L)textlines:=L|@textlinesfor C in @clients do

{C newline(L)}end

endend

Figure 11.14: The original ChatServer class

Each client runs independently as soon as it is created. Its window is associatedwith a thread. Whenever the Send button is pressed, the entered text is sent tothe server, which sends it to the other clients. The server is an active object, sothat clients can enter text concurrently.

Packaging into software components

The first step is to make the client and server into separate software components,which we call functors (see Section 3.9). Functors may need other modules inorder to be instantiated. For example, the client needs the QTk module for itsgraphic user interface. In a distributed program, some of these modules may betied to one process. We call such modules resources. A module is a resource whenit uses information that is only available in one process. The graphics modulesTk and QTk are resources because they access the screen, keyboard, and mouse.Many System modules are resources. For example, the modules Open and Pickle

are resources because they access the file system.The ChatClient class is made into a functor by listing the resources it needs

(see Figure 11.16). The client needs three resources: QTk, the graphic inter-face, Application , which is needed to quit the application, and Distribution ,which defines the NewActive procedure. NewActive is the same as New exceptthat it makes the object an active object. We will see why this is important later


DraftJune 10, 2002


class ChatClientattr

texthandle

meth init(N S C)TH EHQuit= proc {$ W} {W close} {Application.exit 0} endA=proc {$}

{S newline(N#":"#{EH get($)}#"\n")}{EH set(nil)}

endD=td(glue:nswe

text(handle:TH tdscrollbar: truewidth:40 height:10 glue:nswe)

lr(glue:nswebutton(text:"Send" glue:we action:A)entry(handle:EH glue:nswe)button(text:"Quit" glue:we action:{Quit W})))

W={Build td(D title:N#" ´ s Chat Room")}in

{EH bind(event:"<Return>" action:A)}{W show}thread T in

{S newclient(C T)}for L in T do

{TH insert( ´ end ´ L)}end{TH see( ´ end ´ )}

endtexthandle:=TH

end

meth newline(L){@texthandle insert( ´ end ´ L)}{@texthandle see( ´ end ´ )}

endend

Figure 11.15: The original ChatClient class


DraftJune 10, 2002


functorimport QTk Application Distribution(newActive:NewActive)export

server:Sreconnect:R

defineclass ChatClient

...endC={NewActive ChatClient init("Peter" S C)}proc {R}

{S newclient(C T)}{S newline("*** Reconnecting to new server...\n")}for L in T do {C newline(L)} end

endend

Figure 11.16: The client functor

functorexport

server:Sdefine

class ChatServer...

endS={New ChatServer init}

end

Figure 11.17: The server functor

on. The functor also defines a zero-argument procedure R, called the reconnectionprocedure. This procedure is called whenever the client changes servers. In thecase of the chat room, it registers the client to the new server and informs allusers that it has connected to a new server.

Making the ChatServer class into a functor is easier because it does not needany resources–the import clause is absent (see Figure 11.17).

Figures 11.16 and 11.17 show the source code of the two functors thus ob-tained. We put this source code into the files ChatServer.oz and ChatClient.oz

and compile them:

% ozc -c ChatClient.oz

% ozc -c ChatServer.oz

This creates two compiled functors and stores them in ChatClient.ozf andChatServer.ozf. We will use these compiled functors as inputs to the client/servertool.


DraftJune 10, 2002


Using the client/server

To start a distributed server, we call dserver as follows:

% dserver --in=ChatServer.ozf

--publish=/norge/staff/pvr/public_html/chat &

This takes the compiled functor ChatServer.ozf, starts a server, and publishesan entry point in the file /norge/staff/pvr/public_html/chat. Because of theWeb infrastructure, this file is accessible world-wide through an URL. To start aclient, we call dclient as follows:

% dclient --in=ChatClient.ozf

--connect=http://www.info.ucl.ac.be/~pvr/chat &

This takes the compiled functor ChatClient.ozf, finds the server at the givenURL (we suppose that it corresponds to the file used by the server), and startsup a client.

11.6 Transactional object store

This section introduces a transactional object store as an example of a useful fault-tolerance abstraction. A transactional object store is a set of objects that can beupdated by means of transactions (see Section 7.4.10). Most often, we want ourtransactions to commit. But because an abort can occur, the application has tohandle this case. The concept of transaction is therefore a uniform way to handleabnormal situations. In the transactional store, an abort can have an internal orexternal origin:

• Because of an internal cause, i.e., a conflict: two transactions try to modifythe same object simultaneously. There are two possible ways to executethis. In the optimistic case, the transactions do not wait, but one may beaborted if there is a conflict. In the pessimistic case, one of the transactionswill wait until the other is finished. There is no conflict, but performancemay be reduced.

• Because of an external cause, i.e., a failure: during the execution of a trans-action, there is a crash of one or more of the processes hosting the store.The store has to reorganize itself internally before it can accept new transac-tions. It may have to abort some transactions that were in progress duringthe failure.

Section 11.6.1 gives a scenario, a collaborative graphic editor, that can be imple-mented with the transactional store. Section 11.6.2 gives an example of how touse the transactional store. We give the full code for a small, but interesting col-laborative application: a fault-tolerant distributed event counter. Section 11.6.3


DraftJune 10, 2002

11.6 Transactional object store 687

All users manipulate the same drawing

All users have instantaneous response time

-

-Specification

User

User

UserUser

User

Networkwith

performance

unpredictable

Figure 11.18: A collaborative graphic editor

explains the transactional store’s fundamental properties and gives the appli-cation interface. The transactional store is completely implemented using thereflective failure detection provided by the Fault module. It is available as partof the MOGUL library in the Mozart system [109].

11.6.1 A scenario

We present the thoughts underlying the design of a collaborative graphic editor.This shows that the transactional store is a natural abstraction that solves anatural problem.

A collaborative graphic editor

Assume we want to build a collaborative graphic editor. Figure 11.18 shows whatwe want to achieve. Two or more people on opposite parts of the globe can worktogether at the same time on the same drawing. The problem is, the network canbe bothersome. Transmission time and bandwidth between antipodes can runfrom tenths of second to seconds, even minutes if there is congestion. How canwe minimize the bother? A good approach is to first try for the best solution andsee what this leads to. Assume each user can edit with immediate feedback. If thenetwork is slow, then users do not see the edit right away, but after a delay. They


DraftJune 10, 2002


might have already edited the object in question, leading to a conflict. There aremany ways to resolve conflicts. For a graphic editor, the simplest seems to letone user’s changes “win”, or become the globally accepted ones, and the otheruser’s changes “lose”, or disappear. The losing user has done some wasted work.We call this speculative work.

Speculative work can arise even if there are no conflicts. This is more subtle.Assume that a user has made changes to an object, and then no longer uses thatobject. Making the changes, which have been accepted, globally visible takestime. During this time, if another user starts using these same objects, thenhe will be doing speculative work if he does not have the most recent objectstate. When the most recent state arrives, then the work has to be redone. Ifthe user is human, then this can be a bother. If the user is a program that hasprogrammatically changed the objects, then this is no bother: the program issimply run again on the new state.

Speculative work leads naturally to the concept of transaction. For our pur-poses, a transaction is an operation that is atomic and speculative. Atomic meansthat when observed, either it has finished or it has not started yet. No interme-diate states can be observed. This does not mean that they do not exist; theyof course exist if the transaction has to do much work. But hiding them makesthe program much easier to reason about. Speculative means that the transactioncan do wasted work, i.e., it will sometimes be canceled while it is still working.This cancellation has to be complete, i.e., no trace of the work should remain.

The concept of transaction is actually much richer than this. There are manydifferent kinds of transactions with many different properties. Bernstein et al [15]and Gray and Reuter [59] give good introductions to transactions and the theoryunderlying them.

When a user wants to do an operation, it invokes a transaction. The systemtakes care of communicating with the other users, and eventually it comes backto the invoker with one of two messages: commit or abort. If the message is abort,then the system cancels all local operations. All the work was speculative. If themessage is commit, then the local operations become globally accepted and arebroadcasted to the other users. Even with a commit there can be speculativework, as we explained above.

The abstraction behind the editor

As the previous scenario shows, a good abstraction for collaborative work wouldlet users share state and interact with it by means of transactions.

We again try for the best solution. The state should be a set of objects,defined in exactly the same way as any objects in the language, i.e., by means ofclasses, inheritance, and instantiation. Let us call this set a transactional store. Atransaction is then a program fragment that calls objects in a transactional store.Now we run into an inherent limitation: the system has to be able to cancel thework that the objects have done, without it being observable from outside. This


DraftJune 10, 2002


means that an object can reference another object in the transactional store,but it cannot reference any stateful entity outside of the transactional store.How do we get around this limitation? There is a simple technique: defineoperations, called notifications, that execute whenever a transaction commitsor aborts. Notifications can reference state outside of the transactional store,because they themselves are never canceled.

We can use the transactional store to implement the graphic editor. Assumethat the editor’s screen cannot be part of a transactional store, because it’s effectscannot be undone. Each graphic entity on the screen is represented internallyby an object in the transactional store. For example, a rectangle’s object couldcontain its coordinates, its border width, its border color, and its fill color. Whena user changes the rectangle’s appearance, the graphic editor invokes a trans-action. If the transaction is committed, then all other users execute a commitnotification. The notification updates their screens. If notifications did not exist,there would be no way for their screens to get updated, since the only other wayto access object state is by a transaction. If the transaction is aborted, then theoriginal user executes an abort notification. This restores his screen to its originalstate.

All this may seem complicated, but as we shall see it takes only a few lines ofcode. What’s more, the transactional store can take advantage of the redundancythat comes from having more than one user to provide fault tolerance. Withcareful design, it can survive any number of user crashes, as long as at least oneuser is alive at any given moment.

Making it precise

A transactional object store consists of a set of objects that is replicated on severaloperating system processes and that can be updated by means of transactions.The GlobalStore module implements the transactional store abstraction andis available in the MOGUL library. A user is any computation that is part of aprocess running the Mozart system. A user referencing a transactional store canaccess and update its objects. Users can communicate and collaborate by sharinga transactional store.

Store objects can reference other store objects, but they cannot reference anyunbound variables nor any state outside of the store. There are two ways for astore to interact with the external world:

• Through transactions. A transaction is a procedure that is called at a client,and that has two results: commit or abort. Transactions can invoke storeobjects but must not modify state external to the store. A commit meansthe transaction’s results become atomically visible at all users. An abortmeans the transaction has no visible results at all. The system does itsbest to make transactions commit, but due to conflicts or failures, theysometimes have to abort. Since transactions can abort at any time or be


DraftJune 10, 2

002


re-executed at any time (speculative execution), the system is careful tocorrectly modify the external state.

• Through notifications. A notification is a procedure that is called at aclient if some condition is satisfied, e.g., an object state is updated througha committed transaction. Notifications can modify state external to thestore.

11.6.2 Hands-on introduction

Let us build a very small collaborative tool using the transactional store: a faulttolerant event counter. Any user can increment the counter at any time andnew values of the counter are displayed everywhere. The user interface is simple:every user sees a window with a button that contains the current counter value.Clicking on the button increments the counter globally.

We will build this counter in several steps. We start with a single-user versionthat we subsequently make publicly-accessible and distributed. We then put thisversion in the transactional store. Finally, we add a few details to make this intoa nice application: doing recovery properly, cleanly disconnecting clients, andpersistence.

A counter

We start by showing a simple counter with graphical display:

class Counterattr imeth init i:=0 endmeth inc i:=@i+1 endmeth get(X) @i=X end

endC={New Counter init}

Hproc {A} X in {C inc} {C get(X)} {H set(X)} endW={Build

td(title:"Counter"button(text:nil action:A handle:H glue:nswe))}

{W show}

This counter can only be used by one person.

An open distributed counter

To make the counter usable by several people, let us separate the counter partand display parts and put them into separate functors. The counter part is like


DraftJune 10, 2002


a server. It must be started first to create the counter object and offer a ticketfor it:

functorimport Pickle Connection

defineclass Counter

attr imeth init i:=0 endmeth inc i:=@i+1 endmeth get(X) @i=X end

endC={NewStat Counter init}{Offer C ´ ctr.tick ´ }

end

Note that C is a stationary object (defined with NewStat ). The display part islike a client. It can be started up any number of times after the server has started.It takes the ticket and creates the display:

functorimport Pickle Connection QTk

defineC={Take ´ ctr.tick ´ }Hproc {A} X in {C inc} {C get(X)} {H set(X)} endW={Build


{W show}end

The two parts are connected through the common file ´ ctr.tick ´ which containsthe ticket. This version has the disadvantage that the client displays are notupdated when the counter is updated. To fix this problem, we could add a list ofclients in the server, and call all of them whenever the object state is updated.Instead of doing this, let us use the transactional store. This fixes the updateproblem and in addition the counter becomes fault tolerant.

A transactional counter

Let us put the counter object in a transactional store. We first initialize theapplication by defining the object and creating the transactional store:

functorimport Pickle Connection GlobalStore Module

defineclass Counter

attr imeth init i:=0 end


DraftJune 10, 2002


meth inc i:=@i+1 endmeth get(X) @i=X end

end

NewObjBLS={GlobalStore.newStore}LS={BLS newLocal(Module user1 NewObj $ _)}

C={NewObj Counter init}CN={LS saveobj(C $)}{Offer LS#CN ´ ctr.tick ´ }

end

This makes both a store reference and the counter accessible by a ticket in thefile ctr.tick. Now we give a program that adds an additional site to an existingtransactional store and creates a display for the counter:

functorimport Pickle Connection Module QTk

defineH AW={Build


{W show}

BLS#CN={Take ´ ctr.tick ´ }LS={BLS newLocal(Module userN _ $ _)}C={LS loadobj(CN $)}proc {A} TId X in

{LS trans( proc {$ X1} {C inc} {C get(X1)} end X TId)}if {LS checktrans(TId $ _)}==commit then {H set(X)} end

end{LS setnotifyupdate(

proc {$ C} X in {C get(X)} {H set(X)} end )}end

Remark that the second functor does not need to import the GlobalStore mod-ule; its functionality is automatically transferred when the local store LS is cre-ated. This example shows how the transactional store is interfaced with the restof the system:

• Information goes in the store by means of transactions. The procedure A

is called when the button is clicked. It creates a transaction that incre-ments C and reads its value. If the transaction commits (checked with {LS

checktrans(...)} ) then its output is displayed locally.

• Information comes out of the store by means of notifications. A notificationis called whenever a particular event happens. The setnotifyupdate


DraftJune 10, 2002


method installs a procedure that is called whenever the state of an objectis updated.

Example

To use the transactional counter, first compile the above two functors into stan-dalone programs, newctr and ctr. Run newctr once to create a new transactionalcounter. Then run ctr each time a new client is needed. This is a robust counter:any of these processes can be killed at any time. There is no special “server” pro-cess that keeps the counter state; all processes are equivalent. As long as at leastone survives, the counter survives.

Recovery (keeping the ticket alive)

The problem with the above program is that the ticket refers to one particular site,namely the site that originally created the transactional store. What happens ifthis site crashes? Then the ticket is no longer valid, so no new clients can beadded. A simple fix to this problem is to create a new ticket whenever a sitecrashes. We use the setnotifyrecovery to implement this. Add the followingto the ctr program:

{LS setnotifyrecovery(proc {$} {Offer LS#CN ´ ctr.tick ´ } end )}

Here CN is the constant that refers to the counter object. This notification pro-cedure is called by exactly one of the surviving clients whenever there is a crash.

This problem is a special case of a general phase of fault tolerance calledrecovery. When a site fails, the system reconfigures itself to be correct again.Creating a new ticket when an old one is stale is just one example of recovery.Another example is when the system must guarantee that there are always somefixed number (e.g., two) of working sites. When a site dies, then part of therecovery consists in creating a new site to replace it.

Disconnecting clients

So far, the only way for a client to leave the transactional store is by crashing. Weneed to extend the application so that a client can also leave without crashing.We do this by adding a Quit button to the ctr program:

H A QW={Build

td(title:"Counter"lr(button(text:nil action:A handle:H glue:nswe)

button(text:"Quit" action:Q glue:ns) glue:nswe))}{W show}proc {Q}

{LS disconnect}{Application.exit 0}


Draft

June 10, 2002


end

The quit action disconnects the local store and exits the counter client. Themodule Application should be added to ctr’s import list.

Making it persistent

- Modify the example to add persistence: a special user stores

counter values with Pickle. When the store starts up, it first

looks at the pickled value before starting with 1. If the store

crashes completely, then it can be restarted. The restart will

be coherent as long as the pickled value comes from a committed

transaction that reads the store’s state.

- Property of "seedability": can be restarted at any time, due

to persistence will continue to work correctly.

Extending the functionality

- Modify the example so that clicking on Inc increments the

counter AND displays the name of the user who clicked on the Inc.

- Modify the example to make a log of which users clicked on Inc.

11.6.3 Application interface

The transactional store can be seen as a set of objects. The application interfacehas four parts:

• Creation. Store and object creation, connection and disconnection of newclients.

• Transactions. Transactional update of objects in the store.

• Notifications. The store can call stateful entities outside of itself.

• Migration. The store can move. This is in fact an optimization, since thestore is decentralized.

We explain a few of the most important operations for each part. Many moreoperations are available in the full GlobalStore module. For more information,please see the module documentation.

Creation

In each instance of the transactional store, we distinguish the global store andthe local stores. The global store is the high-level view of the whole transactional


Draft

June 10, 2002


store, considered as a set of objects existing on a set of sites. A local store is thepart of the global store visible on a particular site.

LS={GlobalStore.newStore}

Create a new global store, with a single local representative LS. The reference LS

can only be used to create new local stores. Any site can create new local storesby calling LS with newLocal , as explained below.

{LS newLocal(Module UserName ?NewObj ?LS1 ?MoveHere}

Given a local store LS from another site, create a new local store LS1 on thecurrent site. Module is the current site’s module manager, i.e., it references themodule Module . UserName is an arbitrary atom used to identify the local store.In addition to creating LS1, this also defines NewObj, for creating new objects inLS1, and MoveHere , for migrating to the current site. The LS1 reference can beused like LS to create more local stores, and so forth recursively.

X={NewObj Class Init}

Create a new object X in the global store. Class may not reference resources,unbound variables, or any stateful entities external to the global store. This callis synchronous, i.e., it returns after the new object exists in all local stores.

In the current implementation, object references cannot be passed transpar-ently from one local store to another. A translation is needed to and from a localobject reference to a global object reference. This translation is provided by thesaveobj and loadobj operations explained below.

{LS saveobj(Obj ?N)}

Create a globally-unique name N for the global store object Obj . This name canbe passed to any local store site.

{LS loadobj(N ?Obj)}

Return the local store object corresponding to the name N. The name must havebeen created with saveobj .

Transactions

Transactions are the only way to modify objects in a global store. All transactionsinitiated on one site are executed sequentially. Transactions between differentsites can be concurrent.

{LS trans(T ?Out ?TId)}

Initiate a transaction T and returns a transaction identifier TId . The transactionT is a one-argument procedure. If the transaction commits, then T’s argument isbound to Out . Inside T, global store objects may be called, but no state externalto the global store may be changed.

{LS checktrans(TId ?Res ?N)}


DraftJune 10, 2002


Wait until the transaction commits or aborts. Res is bound to the atoms commit

or abort accordingly. N is bound to the number of times that the transaction isattempted (it can be more than 1 due to network delays).

Notifications

Notifications are how a global store can effect the world outside of itself. Execu-tion of notification procedures is serialized with committed transactions on eachsite.

{LS setnotifyupdate(P)}

Add a one-argument notification procedure P to the local store on the currentsite. If an object X is successfully modified by a committed transaction on anyother site, then {P X} is executed. There are no restrictions on P except that itmay not initiate transactions.

{LS setnotifyrecovery(P)}

Add a zero-argument notification procedure P to the local store on the currentsite. This should be done on all sites of the global store. Then, if there is a failureof any local store site, exactly one of the surviving sites will execute P after theglobal store reorganizes itself internally. There are no restrictions on P exceptthat it may not initiate transactions.

Migration

Strictly speaking, migration is not needed since each transactional store is com-pletely decentralized with no single point of failure. For performance reasons,however, a transactional store is organized internally in a client/server structure.This is not a limitation, since if the server site fails, then the clients will electanother server. Migration forces the server to move to the current site. This isjust a performance optimization; it has no effect on the store’s functionality. It isuseful when it is known that the transactional store is no longer needed on someother sites.

{MoveHere Module ?LS P}

Move the transactional store to the current site, i.e., where MoveHere is executed.Then returns a reference LS to a local store on the current site and executeszero-argument procedure P on the current site. Module is the current modulemanager.

11.6.4 Basic properties

The transactional store has the following properties:

• Coherence. The store maintains coherence between the objects. That is,the objects’ states are always eventually the same.


DraftJune 10, 2002

11.7 Exercises 697

• Transactional interface with concurrency control. A user invokesobjects by initiating a transaction, which calls the objects. To supportlong-lived transactions, a “retrans” operation is allowed inside a transaction.Nested transactions are not supported.

• Dynamic. Users can connect to or disconnect from a transactional storedynamically. This adds or removes processes from the transactional store.

• Flexible synchronization. It is possible to use the store in a completelyasynchronous manner, so that a user’s object updates are seen instanta-neously by that user without waiting for the network. This implies a pos-sible speculative execution, which is completely managed by the store. Onthe other hand, completely synchronous operation is possible as well.

• Lightweight. The transactional protocol is optimized for the common casewhen there are no failures. The implementation uses a simple lock serverthat handles distributed object locking. Updating the state of one objectrequires one internal round trip message to the lock server followed by onebroadcast message to all the clients, if the transaction was successful.

• Fault tolerance without persistence. The transactional store toleratesany number of user failures, as long as it exists on at least one process.The transactional store is completely detached from any file system: duringoperation it never needs to read or write from a file system.

• Decentralized. The transactional store has no single point of failure. Aslong as it exists on at least one process, it will work correctly. If the internallock server crashes, then a new lock server is elected. This is invisible tousers of the transactional store.

• Migration. As a consequence of its decentralized nature, the store can mi-grate without dependencies, i.e., the migration depends on no fixed process.

Limitations of the current implementation: one active transaction per client atany instant, no nested transactions, no support for network partitioning.

11.7 Exercises

1. Implementing network awareness. Explain exactly what happens inthe network (what messages are sent and when) during the execution ofthe distributed lexical scoping example given in Section 11.2. Base yourexplanation on the distributed algorithms explained in Section 11.3.4.

2. Fault-tolerant chat room. Reimplement the chat room of Section 11.5.3using the transactional object store of Section 11.6.


DraftJune 10, 2002



DraftJune 10, 2002

Chapter 12

Constraint Programming(incomplete)

“Plans within plans within plans within plans.”– Dune, Frank Herbert (1920–1986)

“Toward the end of the thirteenth century, Ramon Llull (RaimundoLulio or Raymond Lully) invented the thinking machine. [...] Thecircumstances and objectives of this machine no longer interest us,but its guiding principle–the methodical application of chance to theresolution of a problem–still does.”– Ramon Llull’s Thinking Machine, Jorge Luis Borges (1899–1986)

Constraint programming consists of a set of techniques for solving constraintsatisfaction problems. A constraint satisfaction problem, or CSP, consists of a setof constraints on a set of variables. A constraint, in this setting, is simply a logicalrelation, such as “X is less than Y” or “X is a multiple of 3”. The first problem isto find whether there exists a solution, without necessarily constructing it. Thesecond problem is to find one or all solutions.

A CSP can always be solved with brute force search. All possible values ofall variables are enumerated and each is checked to see whether it is a solution.Except in very small problems, the number of candidates is usually too largeto enumerate them all. Constraint programming has developed “smart” waysto solve CSPs, which greatly reduces the amount of search needed. For manyproblems, search can be greatly reduced but not eliminated. This is sufficientto solve many practical problems. Solving CSPs is related to deep questionsof intractability. Problems that are inherently intractable will always need somesearch. The hope of constraint programming is that, for the problems that interestus, the search component can be reduced to an acceptable level.

Constraint programming is qualitatively different from the other programmingparadigms that we have seen, such as declarative, object-oriented, and concurrentprogramming. Compared to these paradigms, constraint programming is much


DraftJune 10, 2002

700 Constraint Programming (incomplete)

closer to the ideal of declarative programming: to say what we want withoutsaying how to achieve it.

Section 3.4.6 gives a tree drawing algorithm that calculates the positions of thenodes of a tree according to five constraints that were picked for esthetic reasons.We can also solve the problem with constraint programming. The program ismuch simpler, since it just consists of the constraints themselves.

12.1 Propagate and search

This chapter presents a general approach for tackling CSPs called propagate-and-search or propagate-and-distribute. In this section, we introduce the basic ideasunderlying this approach and give a simple example. Section 12.3 shows how thecomputation model is extended to support this approach and how to programwith the extended model.

12.1.1 Basic ideas

The propagate-and-search approach is based on three important ideas:

1. Keep partial information. During the calculation, we might have partialinformation about a solution (such as, “in any solution, X is greater than100”). We keep as much of this information as possible.

2. Use local deduction. Each of the constraints uses the partial informationto deduce more information. For example, combining the constraint “X isless than Y” and the partial information “X is greater than 100”, we candeduce that ”Y is greater than 101” (assuming Y is an integer).

3. Do controlled search. When no more local deductions can be done, then wehave to search. The idea is to search as little as possible. We will do justa small search step and then we will try to do local deduction again. Asearch step consists in splitting a CSP P into two new problems, (P ∧ C)and (P ∧ ¬C), where C is a new constraint. Since each new problemhas an additional constraint, it can do new local deductions. To find thesolutions of P , it is enough to take the union of the solutions to the twonew problems. The choice of C is extremely important. A well-chosen Cwill lead to a solution in just a few search steps.

Calculating with partial information

The first part of constraint programming is calculating with partial information,namely keeping partial information and doing local deduction on it. We givean example to show how this works, using intervals of integers. Assume thatx and y measure the sides of a rectangular field of agricultural land in integralmeters. We only have approximations to x and y. Assume that 90 ≤ x ≤ 110


DraftJune 10, 2002

12.1 Propagate and search 701

and 48 ≤ y ≤ 53. Now we would like to calculate with this partial information.For example, is the area of the field bigger than 4000 square meters? This is easyto do with constraint programming. We first declare what we know about x andy:

declare X Y inX::90#110Y::48#53

The notation X::90#110 means x ∈ {90, 91, ..., 110}. Now let us calculate withthis information. With constraint programming, xy > 4000 will return with trueimmediately:1

declare A inA::0#10000A=:X*Y{Browse A>:4000} % Displays 1

We can also display the area directly:

{Browse A} % Displays 4320#5830

From this we know the area must be in the range from 4320 to 5830 square meters.The statement A=:X*Y does a constraint multiplication. Technically, it is calleda propagator: it looks at its arguments a, x, and y, and propagates informationbetween them. In this case, the propagation is simple: the minimal value of ais updated to 90× 48 (which is 4320) and the maximal value of a is updated to110× 53 (which is 5830). Note that we have to give the initial information abouta, for example that it is in the range from 0 to 10000. If we do not give thisinformation, the constraint multiplication A=:X*Y will suspend.

Now let us add some more information about x and y and see what we candeduce from it. Assume we know that the difference x− 2y is exactly 11 meters.We know this by fitting a rope to the y side. Passing the rope twice on the x sideleaves 11 meters. What can we deduce from this fact? Add the constraint:

X-2*Y=:11

Technically, this new constraint is also a propagator. It does a local deductionwith the information we know about x and y. The browser display is automat-ically updated to 5136#5341 . This considerably increases the accuracy of ourmeasurement: we know the area must be from 5136 to 5341 square meters. Whatdo we know about x and y? We can display them:

{Browse X}{Browse Y}

This displays 107#109 for x and 48#49 for y. This is a very simple example ofcalculating with partial information, but it can already be quite useful.

1The program fragment will display the integer 1, which means true. The boolean is givenas an integer because we often need to do calculations with it.


DraftJune 10, 2

002


12.1.2 Example

We now look at an example of a complete constraint program, to see how propagate-and-search actually works. Consider the following problem:

How can I make a rectangle out of 24 unit squares so that the perime-ter is exactly 20?

Say that x and y are the lengths of the rectangle’s sides. This gives two equations:

x · y = 24

2 · (x + y) = 10

We can also add a third equation:

x ≤ y

Strictly speaking, the third equation is not necessary, but including it does noharm (since we can always flip a rectangle over) and it will make the problem’ssolution easier (technically, it reduces the size of the search space). These threeequations are constraints. We call these equations propagators, since we will usethem to make local deductions, i.e., “propagate” partial information about asolution.

To solve the problem, it is useful to start with some information about thevariables. We bound the possible values of the variables. This is not absolutelynecessary, but it is almost always possible and it often makes solving the problemeasier. For our example, assume that X and Y each range from 1 and 9. Thisis reasonable since they are positive and less than 10. This gives two additionalequations:

x ∈ {1, 2, ..., 9}y ∈ {1, 2, ..., 9}

We call these equations basic constraints since they are of the simple form “vari-able in an explicit set”, which can be represented directly in memory.

The initial problem

Now let us start solving the problem. We have three propagators and two basicconstraints. This gives the following situation:

S1 : X*Y=:24 X+Y=:10 X=<:Y || X::1#9 Y::1#9

which we will call the computation space S1. A computation space contains thepropagators and the basic constraints on the problem variables. As in the previousexample, we use the notation X::1#9 to mean x ∈ {1, 2, ..., 9}. We have the threepropagators X*Y=:24 , X+Y=:10 , and X=<:Y . Syntactically, we show that theseare propagators by adding the colon : to their name.


DraftJune 10, 2002

12.1 Propagate and search 703

Local deductions

Each propagator now tries to do local deductions. For example, the propagatorX*Y=:24 notices that since Y is at most 9, that X cannot be 1 or 2. Therefore X isat least 3. It follows that Y is at most 8 (since 3*8=24). The same reasoning can bedone with X and Y reversed. The propagator therefore updates the computationspace:

S1 : X*Y=:24 X+Y=:10 X=<:Y || X::3#8 Y::3#8

Now the propagator X+Y=:10 enters the picture. It notices that since Y cannotbe 2, therefore X cannot be 8. Similarly, Y cannot be 8 either. This gives:

S1 : X*Y=:24 X+Y=:10 X=<:Y || X::3#7 Y::3#7

With this new information, the propagator X*Y=:24 can do more deduction.Since X is at most 7, therefore Y must be at least 4 (because 3*7 is definitely lessthan 24). If Y is at least 4, then X must be at most 6. This gives:

S1 : X*Y=:24 X+Y=:10 X=<:Y || X::4#6 Y::4#6

At this point, none of the propagators sees any opportunities for adding infor-mation. We say that the computation space has become stable. Local deductioncannot add any more information.

Using search

How do we continue? We have to make a guess. Let us guess X=4. To make surethat we do not lose any solutions, we need two computation spaces: one in whichX=4 and another in which X!=4 . This gives:

S2 : X*Y=:24 X+Y=:10 X=<:Y || X=4 Y::4#6

S3 : X*Y=:24 X+Y=:10 X=<:Y || X::5#6 Y::4#6

Each of these computation spaces now has the opportunity to do local deductionsagain. For S2, the local deductions give a value of Y:

S2 : X*Y=:24 X+Y=:10 X=<:Y || X=4 Y=6

At this point, each of the three propagators notices that it is completely solved(it can never add any more information) and therefore removes itself from thecomputation space. We say that the propagators are entailed. This gives:

S2 : (empty) || X=4 Y=6

The result is a solved computation space. It contains the solution X=4 Y=6.Let us see what happens with S3. Propagator X*Y=:24 deduces that X=6

Y=4 is the only possibility consistent with itself (we leave the reasoning to thereader). Then propagator X=<:Y sees that there is no possible solution consistentwith itself. This causes the space to fail:

S3 : (failed)

A failed space has no solution. We conclude that the only solution is X=4 Y=6.


DraftJune 10, 2002


12.1.3 Executing the example

Let us run this example in Mozart. We define the problem by writing a one-argument procedure whose argument is the solution. Running the procedure setsup the basic constraints, the propagators, and selects a distribution strategy.Here is the procedure definition:

proc {Rect ?Sol}sol(X Y)=Sol in

X::1#9 Y::1#9X*Y=:24 X+Y=:10 X=<:Y{FD.distribute ff Sol}

end

The solution is returned as the tuple Sol , which contains the two variables X

and Y. Here X::1#9 and Y::1#9 are the two basic constraints and X*Y=:24 ,X+Y=:10 , and X=<:Y are the three propagators. The FD.distribute call selectsthe distribution strategy. To find the solutions, we pass the procedure to a generalsearch engine:

{Browse {Search.base.all Rect}}

This displays a list of all solutions, namely [sol(4 6)] since there is only one.

All the constraint operations used in this example, namely :: , =: , =<: ,FD.distribute , and Search.base.all , are predefined in the Mozart system.The full constraint programming support of Mozart consists of several dozen op-erations. All of these operations are defined in the constraint-based computationmodel. This model introduces just two new concepts to the stateful concurrentmodel: finite domain constraints (basic constraints like X::1#9 ) and computa-tion spaces. All the richness of constraint programming in Mozart is provided bythis model.

12.1.4 Summary

The fundamental concept used to implement propagate-and-search is the compu-tation space, which contains propagators and basic constraints. Solving a problemalternates two phases. A space first does local deductions with the propagators.When no more local deductions are possible, i.e., the space is stable, then a searchstep is done. In this step, two copies of the space are used. A constraint C isguessed according to a heuristic called the distribution strategy. The basic con-straint C is added to the first copy and not C is added to the second copy. Wethen continue with each copy. The process is continued until all spaces are eithersolved or failed. This gives us all solutions to the problem.


Draft

June 10, 2002

12.2 Programming techniques 705

12.2 Programming techniques

Now that we have seen the basic concepts, let us see how to program with them.A constraint problem is defined by a one-argument procedure. The procedureargument is bound to the solution to the problem. Inside the procedure, next tothe usual language operations, two new kinds of operations are possible:

• Constraints. These specify the relationships between the different parts ofthe problem.

• Specification of the distribution strategy. This specifies how the search treeis to be formed.

In contrast to relational programming (see Chapter 8), there is no explicit creationof choice points (no choice statement). This would be too crude a way to search;what actually happens is that choice points are created dynamically in terms ofthe distribution strategy that is specified.

12.2.1 Example problem

We can see how this works with an example.2 A well-known combinatoric puzzleis the Send-More-Money problem. The problem is to assign digits to letters suchthat the following addition makes sense:

S E N D+ M O R EM O N E Y

There are two conditions: each letter is assigned to a different digit and theleading digits of the numbers are different from zero (S 6= 0 and M 6= 0).

To solve this problem with constraints, the first step is to model the problem,i.e., to set up data structures and constraints that reflect the problem structure.In this problem, it is easy: each digit is a variable and the problem conditions be-come constraints on the variables. There are eight different letters, and thereforeeight variables.

The second step is to define a one-argument procedure that implements thismodel. Figure 12.1 shows one way to define the procedure. The numbered state-ments have the following effects:

1. The solution Sol is a record with one field for every different letter.

2. The fields of Sol are integers in the domain {0, ..., 9}.

3. The fields of Sol are pairwise distinct, i.e., no two have the same value.

4. Since they are leading digits, the values of S and Mare not zero.

2This example is taken from [141].


DraftJune 10, 2

002


proc {SendMoreMoney ?Sol}S E N D M O R Y

inSol=sol(s:S e:E n:N d:D m:M o:O r:R y:Y) %1Sol:::0#9 %2{FD.distinct Sol} %3S\=:0 %4M\=:0

1000*S + 100*E + 10*N + D %5+ 1000*M + 100*O + 10*R + E=: 10000*M + 1000*O + 100*N + 10*E + Y{FD.distribute ff Sol} %6

end

Figure 12.1: Constraint definition of Send-More-Money puzzle

5. All the digits satisfy the equation SEND + MORE = MONEY .

6. The distribution strategy tries the letters according to a first-fail strategy(ff ). This means that the strategy tries first the letter with the leastnumber of possibilities, and with this letter it tries the least value first.

The third step is to solve the problem:

{Browse {SearchAll SendMoreMoney}}

This computes and displays a list of all solutions. Note that this is done in thesame way as search in relational programming (see Chapter 8). This displays:

[sol(d:7 e:5 m:1 n:6 o:0 r:8 s:9 y:2)]

That is, there is just one solution, and it is:

9 5 6 7+ 1 0 8 51 0 6 5 2

That is all there is to it! In practice, things are a bit more complicated:

• Modelizing the problem. Modelizing the problem is not always easy.Sometimes there are different models–which one is best is not always obvi-ous.

• Constraints and distribution strategies. There are many constraintsand distribution strategies to choose from. Which one is best dependsstrongly on the problem.

• Understanding the problem. The first solution to a realistic problem isusually too inefficient. There are many techniques to improve it. For exam-ple: take advantage of problem structure, use redundant constraints, use


DraftJune 10, 2

002


different distribution strategies, use the Explorer (an interactive graphicalsearch tree exploration tool, see [139]).

This chapter explains many examples, principles, and tools in each of these areas.

12.2.2 Palindrome products revisited

In Section 8.2.1, we saw how to find palindrome products with relational pro-gramming. The technique used there takes 45 seconds to find all solutions for 6digit palindromes. Here is a smarter solution that takes advantage of constraints:

proc {Palindrome ?Sol}sol(A)=SolB C X Y Z

inA::0#999999 B::0#999 C::0#999A=:B*CX::0#9 Y::0#9 Z::0#9A=:X*100000+Y*10000+Z*1000+Z*100+Y*10+X{FD.distribute ff [X Y Z]}

end

This takes slightly less than two seconds. We can do even better by realizingthat a palindrome XY ZZY X is always a multiple of 11. That is, XY ZZY X =X · 100001 + Y · 10010 + Z · 1100, which means XY ZZY X/11 = X · 9091 + Y ·910 + Z · 100. Taking advantage of this, we can specify the problem as follows:

proc {Palindrome ?Sol}sol(A)=SolB C X Y Z

inA::0#90909 B::0#90 C::0#999A=:B*CX::0#9 Y::0#9 Z::0#9A=:X*9091+Y*910+Z*100{FD.distribute ff [X Y Z]}

end

This takes slightly less than 0.4 seconds to solve the same problem. What canwe conclude from this simple example? Many things:

• A constraint-based formulation of a combinatoric problem can be muchfaster than a naive generate-and-test formulation. For palindrome product,the constraint solution is more than 100 times faster than the naive solution.

• To make it fast, you also have to take advantage of the problem structure.A little bit of smarts goes a long way. For palindrome product, takingadvantage of the solution being a multiple of 11 makes the program 5 timesfaster.


DraftJune 10, 2002


• A fast solution is not necessarily more complicated than a slow solution.Compare the slow and fast solutions to palindrome product: they are aboutequal in length and ease of understanding.

• Performance can depend strongly on the exact problem formulation. Chang-ing it a little bit can make it much faster or (usually) much slower.

• To write a good specification, you have to understand the operational mean-ing of the constraints as well as the logical meaning. The latter is enoughfor showing correctness, but the former is essential to get good performance.

12.2.3 Drawing trees revisited

With constraint programming we can give a very general form of the tree-drawingalgorithm of Sections 3.4.6 and 4.4.1. Basically, the program just gives the con-straints, in a form very close to Figure 3.16. This is the purest form of definitionaldeclarative programming. The order of calculation is not given and it is not evensure that there are enough numbers to do the calculation. The same program willwork for any constraints. A very similar program can also be used to calculateoptimal tree positionings, according to some criterium.

The constraint-based tree drawing algorithm will find coordinates (x,y) for allnodes to satisfy the following numbered constraints:

1. All nodes are to the right of the y axis (x ≥Scale ).

2. The y position of a node is a function of the level in the tree (y=Scale*L+Scale ).

3. All subtrees of the same parent are separated by a minimal value.

4. All root nodes of subtrees are symmetrical about their parent node.

5. If a node has exactly one child node, then both nodes have the same xcoordinate.

Figure 12.2 gives the function MakeTreeScript , which returns the solution pro-cedure for any tree. The comments give the constraint numbers according to theabove numbering. The function uses several utility routines to do tree traversaland to post constraints at particular places in the tree. We define these utilityroutines below.

The algorithm of Section 3.4.6 will suspend if the wrong calculation order isspecified. There is another way for a tree drawing algorithm to block: namely, ifthere are not enough constraints to specify the problem completely. With search,it is possible to calculate the tree’s position in both of these cases.


DraftJune 10, 2002


Scale=40fun {MakeTreeScript Tree}

proc {$ ?Root}Root={MakePos Tree}{Traverse1 Root 0

proc {$ X Y T L}% 1. X position offset from left edgeX>=:Scale% 2. Y position function of depthY=:Scale*L+Scale

end }{Traverse2 Root 0

proc {$ X Y T AX AY AT BX BY BT L}% 3. Subtrees separated{MaxX AT}+Scale=<:{MinX BT}% 4. Subtrees symmetricAX+BX=:2*X

end }{Traverse4 Root 0

proc {$ X Y T AX AY AT L}% 5. If one child, directly belowX=:AX

end }{FD.distribute ff {MakeList Root $ nil}}

endend

Figure 12.2: Tree drawing algorithm with constraint programming

Example execution

Here is an example execution. First, define the tree:

Tree=tree(... % Give a tree in the same syntax as before

)

Second, define the problem procedure:

TreeScript={MakeTreeScript Tree}

Third, solve the constraint problem:

T={Search.base.one TreeScript}.1

The result of Search.base.one is a one-element list [Sol] if there is at leastone solution, or nil if there is no solution. We assume that there is a solution!We can display the solution in the browser:

{Browse T}

We can also draw this solution to give a nice picture.


DraftJune 10, 2002


Generic tree traversal

The constraint-based tree-drawing algorithm uses several utility routines to postconstraints at well-defined places in the tree. These routines are simple examplesof declarative programs. They are useful for many constraint problems based ontrees.

• Convert the input tree to an internal format that has new fields x and y ,which will contain the positions. With {FD.decl} we initialize these newfields to say that they can hold any integer, but we do not yet know whichone.

fun {MakePos Tree}case Treeof tree(left:L right:R ...) then

{Adjoin Treetree(x:{FD.decl} y:{FD.decl}

left:{MakePos L} right:{MakePos R})}[] leaf then

leafend

end

• Traverse the tree and execute a procedure P at well-defined points in thetree. We define four traversal procedures that each takes three arguments:PosTree , an input tree, Level , an initial level in the tree, and P, a proce-dure to be executed at particular points in the tree. The four proceduresdiffer in where P is executed and what P’s arguments are. Here they are:

– First traversal procedure. Runs {P X Y T Level} on all nodes, where(X,Y) gives the node’s coordinates, T is the subtree at that node, andLevel is the depth in the tree.

proc {Traverse1 PosTree Level P}case PosTreeof tree(x:X y:Y left:L right:R ...) then

{P X Y PosTree Level}{Traverse1 L Level+1 P}{Traverse1 R Level+1 P}

[] leaf thenskip

endend

– Second traversal procedure. Runs {P X Y T AX AY AT BX BY BT

Level} on all nodes with exactly two children. For each node andeach child (A and B), the (X,Y) coordinates and the tree T at the nodeare given.


DraftJune 10, 2002


proc {Traverse2 PosTree Level P}case PosTreeof tree(x:X y:Y

left:L=tree(x:LX y:LY ...)right:R=tree(x:RX y:RY ...) ...) then

{P X Y PosTree LX LY L RX RY R Level}{Traverse2 L Level+1 P}{Traverse2 R Level+1 P}

else skip endend

– Third traversal procedure. Runs {P X Y T SX SY ST Level} on alllinks parent-child.

proc {Traverse3 PosTree Level P}case PosTreeof tree(x:X y:Y left:L=tree(x:LX y:LY ...) ...) then

{P X Y PosTree LX LY L Level}{Traverse3 L Level+1 P}

else skip endcase PosTreeof tree(x:X y:Y right:R=tree(x:RX y:RY ...) ...) then

{P X Y PosTree RX RY R Level}{Traverse3 R Level+1 P}

else skip endend

– Fourth traversal procedure. Runs {P X Y T SX SY ST Level} onall nodes with exactly one child.

proc {Traverse4 PosTree Level P}case PosTreeof tree(x:X y:Y

left:L=tree(x:LX y:LY ...)right:leaf ...) then

{P X Y PosTree LX LY L Level}{Traverse4 L Level+1 P}

[] tree(x:X y:Yleft:leafright:R=tree(x:RX y:RY ...) ...) then

{P X Y PosTree RX RY R Level}{Traverse4 R Level+1 P}

[] tree(left:L right:R ...) then{Traverse4 L Level+1 P}{Traverse4 R Level+1 P}

[] leaf thenskip

endend


DraftJune 10, 2002


• Find the X coordinate of the leftmost node in a tree.

fun {MinX PosTree}case PosTreeof tree(x:X left:leaf ...) then

X[] tree(left:L ...) then

{MinX L}end

end

In analogous fashion, we define a function MaxX that finds the X coordinateof the rightmost node in a tree.

• Give a difference list containing all the X and Y variables in all nodes.

proc {Tree2List PosTree S1 S0}case PosTreeof tree(x:X y:Y left:L right:R ...) then S2 S3 in

S1=X|Y|S2{Tree2List L S2 S3}{Tree2List R S3 S0}

[] leaf thenS1=S0

endend

Experimenting with other constraints

A major advantage of the constraint algorithm is that we can experiment withother constraints and get results immediately. Because search makes the programcomplete, it is not necessary for the programmer to carefully figure out executionthe calculation order. Even stronger, even if an order does not exist, the programwill still find results. For example, here are two nice experiments to do:

• Constrain a minimal distance between nodes at each level, instead of be-tween subtrees. This gives trees that take up less space and still look nice.

• Constrain a minimal distance between each pair of nodes. With branch-and-bound, this should give a very nice solution with reasonable efficiency.

Some experiments can give programs that take a very long time to execute. Thisis because the search space has become very large. This is reverse side of thecoin of the flexibility of constraint programming: there are no guarantees as toexecution time. With problems like tree drawing this is not so important if theconstraints are ones that can be solved directly (e.g., like in the first algorithm).If there is not so much propagation, then it can be slower. If the execution time isvery long, then the problem has to be reformulated. There are many techniquesto improve the performance of a constraint program.


DraftJune 10, 2002

12.3 Constraint-based computation model 713

...S1 S2 Sn

Mutable store

Threads

Constraint store

...

Mutable store

Threads

X=

Constraint store

ST1 ST2 STn

...S1 S2 Sn

Mutable store

Threads

Constraint store

...

Mutable store

Threads

X=

Constraint store

ST1 ST2 STn

...S1 S2 Sn

Mutable store

Threads

Constraint store

...

Mutable store

Threads

X=

Constraint store

ST1 ST2 STn

Top-levelcomputation space

Child space

Child space

W=atom

c1:WY=c1

W=atom

c1:WY=c1

W=atomc1:WY=c1

W=atom

c1:WY=c1

W=atomc1:WY=c1

W=atom

c1:WY=c1

Figure 12.3: Constraint-based computation model

12.3 Constraint-based computation model

The propagate-and-search approach is supported by adding two concepts to thestateful concurrent model: basic constraints and computation spaces. Basicconstraints are a simple generalization of declarative variables in the single-assignment store. Computation spaces are a new ADT.

12.3.1 Basic constraints

12.3.2 Computation spaces

In the previous sections we have seen how to use constraints with built-in dis-tribution strategies. But these distribution strategies are not always sufficient.In this section we explain how computation spaces work and we show how toprogram distribution strategies with them.

Computation spaces are an abstraction that permits the high-level program-ming of search abstractions and deep guard combinators. With computation


DraftJune 10, 2002


spaces, the computation model looks something like Figure 12.3. All the searchabstractions of Chapters 8 and 12 are programmed using spaces. Spaces have theflexibility needed for real-world constraint problems and they can be implementedefficiently: on real-world problems the Mozart implementation using copying andrecomputation is competitive in time and memory use with traditional systemsusing trailing-based backtracking [135].

Spaces are a natural way to integrate search, which has a strong speculativecomponent, into a concurrent system that interacts with its environment. We saythat a part of a search computation is speculative if it is possible to change the finalcomputation by making different choices, so that the result is not changed yet thepart is no longer part of it. That is, the presence of the part is contingent on thechoices made during search, and is not necessary for the final result. Managingspeculative computation is the essence of nondeterministic logic programming.Spaces are compositional, i.e., they can be nested, which is an important property.

This section defines computation spaces, the operations that can be performedon them (see Table 12.1), and gives examples of how to use them to programsearch. The discussion in this section follows the model in [140, 137]. This modelis implemented in the Mozart system [110] and refines the one presented in thearticles [134, 138]. The space abstraction can be made language-independent;[71] describes a C++ implementation of a similar abstraction that provides bothtrailing and copying.

Definition

A computation space is an instance of the stateful concurrent model, with itsthree parts: thread store, constraint store, and mutable store. Since spaces canbe speculative, it is important to be precise about the visibility of variables andtheir bindings. Figure 12.4 gives an example. The general rules for the structureof computation spaces are as follows.

• There is always a top level computation space where threads may interactwith the external world. Because the top level space interacts with theexternal world, its constraint store always remains consistent, that is, eachvariable has at most one binding that never changes once it is made. Athread that tries to add an inconsistent binding to the top level constraintstore will raise a failure exception.

• A thread may create a new computation space. The new space is calleda child space. The current space is the child’s parent space. At any time,there is a tree of computation spaces in which the top level space is theroot. With respect to a given space, a higher one in the tree (closer to theroot) is called an ancestor and a lower one is called a descendant.

• A thread always belongs to exactly one computation space. A variablealways belongs to exactly one computation space.


DraftJune 10, 2002


Space A

Space B

Top Level Space

Anc

esto

rs

X

Ta

TbX=Term

Threads Ta, Tb, and Tc all see●

variable X.

If Tb binds X then Tb & Tc willsee the binding. Ta won’t unlessSpace B is merged into Space A.

●

● This is because child spaces are

become part of their parent store.speculative: they may or may not

Because Space C is speculative,●

only Tc sees Y (Ta and Tb don’t).

Space C

Tc

Y

Des

cend

ants parent

parent

binding

sees

sees

Current space

sees

parent

Figure 12.4: Visibility of variables and bindings in nested spaces

• A thread sees and may access variables belonging to its space as well asto all ancestor spaces. The thread cannot see the variables of descendantspaces.

• A thread cannot see the variables of a child space, unless the child space ismerged with its parent. Space merging is an explicit program operation. Itcauses the child space to disappear and all the child’s content to be addedto the parent space.

• A thread may add bindings to variables visible to it. This means that itmay bind variables belonging to its space or to its ancestor spaces. Thebinding will only be visible in the current space and its descendants. Thatis, the parent space does not see the binding unless the current space ismerged with it.

• If a thread in a child space tries to add an inconsistent binding to its con-straint store, then the space fails.


DraftJune 10, 2002


State of a space

A space is runnable if it or a descendant contains a runnable thread, and blockedotherwise. Let us run all these threads until the space is blocked. Then the spacecan be in one of the following further states:

• The space is stable. This means that no additional bindings done in anancestor can make the space runnable. A stable space can be in four furtherstates:

– The space is succeeded. This means that it contains no choice points.A succeeded space contains a solution to the logic program.

– The space is distributable. This means that the space has one threadthat is suspended on a choice point with two or more alternatives. Aspace can have at most one choice point.

– The space is failed. This is defined in the previous section; it meansthat the space attempted to bind the same variable to two differentvalues. No further execution happens in the space.

– The space is merged. This means that the space has been discarded andits constraint store has been added to a parent. Any further operationon the space is an error. This state is the end of a space’s lifetime.

• The space is not stable. This means that additional bindings done in anancestor can make the space runnable. Being not stable is usually a tem-porary condition due to concurrency. It means that some ancestor spacehas not yet transferred all required information to the space. A space thatstays not stable indefinitely usually means a programmer error.

Programming distribution strategies

Distribution strategies are programmed with spaces in the following way:

• Create the space with the correct program inside.

• Let the program run inside the space. The program will do as much propa-gation as possible, but will eventually block when no more propagation canbe done. At that point, it may possibly have created a choice point.

• Detect when the program can go no further, i.e., when the space has becomestable. In that case, either stop and return with a solution, or continueexecution by forcing the choice point to commit to an alternative.

The next section explains the operations we need for this approach. Then, Sec-tion 12.3.2 gives some examples of how to program search with them. Manyother strategies can be programmed than are shown here; for more informationsee [140, 137].


DraftJune 10, 2

002


Operation DescriptionX={NewSpace P} Return a new computation space.I={Choose N} Create a choice point; returns the chosen alternative.A={Ask X} When the space is stable, returns its status.{Commit X I} Cause the choice point to return with I .C={Clone X} Return an identical copy of space X.{Inject X P} Inject an execution into space X.{Merge X Y} Binds Y to space X’s root variable.

Table 12.1: Primitive operations for computation spaces

Space operations

Now we know enough to define the primitive space operations. There are sevenprincipal ones (see Table 12.1):

• X={NewSpace P} , when given a unary procedure P, creates a new compu-tation space and returns a reference to it. In this space, a fresh variable R,called the root variable, is created and a new thread, and {P R} is invokedin the thread.

• Y={Choose N} first creates in the current space a choice point with N al-ternatives. Then it blocks, waiting for an alternative to be chosen by aCommit operation on the space (see below). The Choose call defines onlythe number of alternatives; it does not specify what to do for any givenalternative. Choose returns with Y=I when alternative 1≤I ≤N is chosen.A maximum of one choice point at a time may exist in a space.

• A={Ask X} asks the space X for its status. As soon as the space becomesstable, A is bound. If X is failed, merged, or succeeded, then Ask returnsfailed , merged , or succeeded . If X is distributable, then it returns al-

ternatives(N) , where N is the number of alternatives.

• {Commit X I} , if X is a distributable space, causes the Choose call in thespace to complete and return I as its result. This may cause the space toresume execution. The integer I must satisfy 1≤I ≤N, where N is the firstargument of the Choose call.

• C={Clone X} , if X is a stable space, creates an identical copy (a clone) of X

and returns a reference to it. This allows both alternatives of a distributablespace to be explored.

• {Inject X P} is similar to space creation except that it uses an existingspace X. It creates a new thread in the space and invokes {P R} in thethread, where R is the space’s root variable. This makes a stable space not


Draft

June 10, 2002


A={Ask X}case A ofalternatives(N) then

{Commit X I}end

...

I={Choose N}case I of

1. Block

5. Synch on alternative(pass I)

of 1 then ...

end...

6. Run alternative[] 2 then ...

......

(pass N)3. Synch on stability

(in parent space)Search strategy

Computation space X

...

4. Calculate alternative

2. Block

...

Figure 12.5: Communication between a space and its distribution strategy

〈statement〉 ::= choice 〈inStatement〉 { ´ [] ´ 〈inStatement〉 } end | ...〈expression〉 ::= choice 〈inExpression〉 { ´ [] ´ 〈inExpression〉 } end | ...

Table 12.2: The choice statement

stable again. Adding constraints to an existing space is necessary for somedistribution strategies such as branch-and-bound and saturation.

• {Merge X Y} binds Y to the root variable of space X and discards the space.

Using spaces

These seven primitive operations are enough to define many distribution strate-gies and deep guard combinators [140, 137]. The basic technique is to use Choose ,Ask , and Commit to communicate between the inside of a space and the distri-bution strategy, which is programmed in the parent space. Figure 12.5 showshow the communication works. Within the space, calling I={Choose N} first in-forms the distribution strategy of the total number of alternatives (N). Then thedistribution strategy picks one (I ) and informs the space. The synchronizationcondition between the inside of the space and the search strategy is stability, i.e.,that there are no more local deductions possible inside the space. We now give afew examples of how to use spaces.

Depth-first search. Figure 12.6 shows how to program depth-first single so-lution search, in the case of binary choice points. This explores the search tree indepth-first manner and returns the first solution it finds. The problem is definedas a unary procedure {Script Sol} that gives a reference to the solution Sol ,just like the examples of Chapter 12. The solution is returned in a one-element


DraftJune 10, 2002


fun {DFE S}case {Ask S}of failed then nil[] succeeded then [S][] alternatives(2) then C={Clone S} in

{Commit S 1}case {DFE S} of nil then {Commit C 2} {DFE C}[] [T] then [T]end

endend

% Given {Script Sol}, returns solution [Sol] or nil:fun {DFS Script}

case {DFE {NewSpace Script}} of nil then nil[] [S] then [{Merge S}]end

end

Figure 12.6: Depth-first single solution search

〈statement〉 ::= dis 〈inStatement〉 then 〈statement〉{ ´ [] ´ 〈inStatement〉 then 〈statement〉 } end | ...

Table 12.3: The dis statement

list as [Sol] . If there is no solution, then nil is returned. In Script , choicepoints are defined with the Choose operation. A naive choice point, one thatselects all its alternatives in order, can be defined as follows:

case {Choose N}of 1 then S1

[] 2 then S2

...[] N then Sn

end

This is exactly the semantics of the choice statement used in Chapter 8, whosesyntax is given in Table 12.2.

Many other search techniques and strategies can be programmed. Some basictechniques are n-ary search, all-solution search, and lazy search. Some basicstrategies are limited discrepancy search, best first search, and branch-and-boundsearch. All these can be combined and parametrized in many ways.


DraftJune 10, 2002


Andorra-style disjunction (the dis statement). The previous exampleshows how to program the choice statement, which is a naive choice point. Thedis statement is an optimization of choice that eliminates failed alternatives andcommits immediately if there is a single remaining alternative. Table 12.3 givesits syntax. The dis statement can be programmed with the space operations asfollows. First encapsulate each guard of the dis statement in a separate space.Then execute each guard until it is stable. Discard all failed guards. Finally,using the Choose operation, create a choice point for the remaining guards. Theexact definition of dis is a bit subtle; see [140, 137] for details.

If dis is important for real programs, then its common cases can be madefaster by using case to do preselection before doing any space operations. Thiswill avoid using spaces in those cases where they are not really needed. In prac-tice, we find that dis is almost never used outside of small examples, so thisoptimization is not worth the effort.

Deep guard combinators. With spaces, it is possible to program constraintcombinators that are both compositional and conservative. This is a powerfulway to combine constraints from different domains and to prototype new con-straints. Some examples of combinators are deep negation, constraint reification,propagation-based disjunction (such as dis ), and deep committed-choice (deepcond ).

12.3.3 Implementing the relational computation model

The relational computation model of Chapter 8 extends the declarative modelwith choice and fail statements and with a Search module to do encapsulatedsearch. All these abilities can be programmed with computation spaces. Finitedomain constraints are not needed.

12.4 Case study: an intelligent minesweeper

As a bigger example of constraint programming, we give an intelligent minesweep-er application. This application is like the popular minesweeper game, which isextended to do deductions about safe moves. The application is an assistant to ahuman player: it does deductions about the playing field (using the same knowl-edge that the human player has) and advises the human player on safe moves(squares with no mine underneath). It is a good example of an application thatis built around constraint programming but that uses many other programmingtechniques as well.


Draft

June 10, 2002

12.4 Case study: an intelligent minesweeper 721

Figure 12.7: The minesweeper application

12.4.1 Rules of the game

The minesweeper is a puzzle game. The playing field is a grid of squares, each ofwhich may hide a mine. The total number of hidden mines is known when thegame starts. The objective is to find out where they are located. The rules aresimple:

1. You uncover a square by clicking on it. If you click on a mine, it explodesand the game is over. Otherwise, a number is displayed in the square givingthe number of mines in the adjacent squares.

2. You can mark a square as a mine by clicking on it with the middle buttonof the mouse. This mark has no effect on the program execution, it justhelps you remember your knowledge about the field.

3. You win the game when all the non-mined squares have been uncovered.To help you, a counter in the window indicates the number of squares stillto be uncovered.

Figure 12.7 shows the user interface. In this case the playing field has dimensions10×10 and contains 20 mines. We can see that 39 squares have been uncoveredand 8 squares have been marked with a cross. A message tells us that 41 squaresare still to be uncovered.

The checkbuttons under the mine field reveal an important feature of ourminesweeper application: the digital assistant. A digital assistant is a piece ofsoftware that helps a user to complete a task. Our assistant consists of intelligentagents that either infer new knowledge about the hidden minefield or automatical-ly play squares that are known to be safe. We define an agent as an independentactivity that can communicate with its environment. We call it intelligent be-cause of its ability to do nontrivial deductions about the state of the minefield.


DraftJune 10, 2002


Game

Userinterface

Gamestatus

Human

Autoplay propagationZero

+ searchConstraints

Passive object

Active object

Message

Dataflow link

Humanplayer

play(I J) bind

assistantDigital

set(B) set(B)

play(I J)

set(B)search(...)

bind

Figure 12.8: Architecture of the minesweeper application

In the figure, the digital digital assistant is active. It has inferred, for instance,that the square at the tenth row and fourth column is a mine. The assistant alsoallows us to perform a local search by right-clicking on the square at the fifth rowand sixth column. If we do this, it will infer that the clicked square is not a mine,and the autoplaying agent will uncover that square. An important property ofthe digital assistant is that it never infers wrong knowledge.

12.4.2 The architecture and implementation

Figure 12.8 shows the architecture in detail, with all the messages passed betweenthe different parts. An interesting feature of this application is that many of theparts communicate through a dataflow link. For example, the display module hasone thread per game square, that waits for the game field to become bound andthen displays it.

We present the code in three steps. The whole program including the dig-ital assistant and the user interface is less than 400 lines long. Section 12.4.3shows how the game itself is programmed. This consists of the classes GameandGameStatus . Section 12.4.4 shows the construction of the graphical user inter-face. Finally, Section 12.4.5 shows the digital assistant and explains each of itsinference modules.


DraftJune 10, 2002


The corresponding value of G.field is

mat(row(_ _ _ mine(marked) safe(1))row(_ _ _ safe(3) safe(2))row(_ _ _ safe(_) mine(exploded))row(_ _ _ _ _)row(_ _ _ _ _))

Figure 12.9: An example for the value of G.field

12.4.3 The game

The Gameclass

The class Gameimplements the minesweeper game and gathers knowledge aboutit. It is specified by the following methods and feature:

G={New Game init(NR NC NM)} creates a new game Gwhose playing field hasNRrows, NCcolumns and contains NMmines.

{G play(I J)} uncovers the square at row I and column J . The result is givenin G.field .

G.field is a feature of the object G. A feature is a stateless component thatis accessed by the feature name (here field ), which is declared by thekeyword feat .

The value of G.field is a matrix whose elements represent the knowl-edge about their corresponding square in the mine field. A matrix withn rows and m columns is a record mat( row1 row2 · · · rown) (with features1, 2, . . . , n) where each rowi is a record row( x1 x2 · · ·xm) (with features1, 2, . . . , m). The module Matrix has been defined to manipulate themeasily.

The possible values of G.field.I.J are explained with the example inFigure 12.9. When the game starts, all the squares are unknown. Forinstance, G.field.1.1 is bound to _, a notation that stands for local

X in X end , i.e., an unbound variable. The possible future bindings areshown in the following table, with their meaning.

Bound value Knowledge about the corresponding square(a) safe(2) safe square with two adjacent mines(b) mine(exploded) mine that has exploded(c) safe(_) safe square (unknown number of adjacent mines)(d) mine(marked) mine that is marked


DraftJune 10, 2002


% Make random list of length N with N1 1’s and N-N1 0’sfun {Random01 N N1} ... end

% Sum the elements of list Lfun {SumList L} {List.foldR L Number. ´ +´ 0} end

% A Game object hides the actual mine field of a game% and retains information from played elementsclass Game

attr BinField field nrows ncols nmines

meth init(NR NC NM)@nrows=NR@ncols=NC@nmines=NM@BinField={Matrix.new NR NC}{Matrix.flatten @BinField}={Random01 NR*NC NM}@field={Matrix.new NR NC}

end

meth play(I J)case (@BinField).I.J of 0 then

N={SumList{Matrix.neighborList @BinField I J}}

in(@field).I.J=safe(N)

[] 1 then(@field).I.J=mine(exploded)

endend

end

Figure 12.10: The class Gamewhich implements the game

The values (a) and (b) are bound by the method play of the game, while(c) and (d) are bound by inference objects. Note that the inferred valuecannot be mistaken.

The code in Figure 12.10 precisely states what we just explained. A Gameobjecthas features nrows , ncols , nmines , which are the number of rows, columnsand mines of the game field, respectively. The feature BinField is the binarymatrix that represents the mine field: values 1 and 0 respectively indicate thepresence and the absence of a mine. The capital initial states that BinField isonly accessible within the lexical scope of the class.

The method init initializes the object’s features. The matrix self .BinField

is filled with random values, while self .field is a matrix with unbound values.


DraftJune 10, 2

002


still(3)

won

still(2)

still(1)

lost

{P won}

{P still(1)}

{P still(2)}

{P still(3)}

{P lost}

(initial state)

(final states)

State Descriptionstill(N) N squares are

still to be un-covered

won the player haswon

lost the player haslost

Figure 12.11: The state diagram of a GameStatus object

The method play updates self .field with one of the values shown above. Thecall to Matrix.neighborList returns the list of elements in self .BinField

neighboring self .BinField.I.J , and the number of 1 among them is given bytheir sum.

The GameStatus class

We show a first use of G.field with the class GameStatus . A GameStatus

object implements a finite-state automaton that calls a given unary procedurewith each state it reaches. Figure 12.11 shows a state diagram for such an object,along with a table describing its states. The latters describe a game field. Everystate transition calls a procedure P with the new state, in such a way that theprocedure calls are done sequencially.

The class defines only one method:

GS={New GameStatus init(G P)} creates a GameStatus object from a gameG and a unary procedure P.

The code of class GameStatus is given in Figure 12.12. The attribute state

contains the state of the automaton. The method init initializes state , calls P

for the first time, and creates a thread for each element in G.field . This threadwaits until the value of the element changes the automaton’s state and does thecorresponding transition. The construct

lock ... end


DraftJune 10, 2

002


% A GameStatus object calls a procedure P% with each state of the game Gclass GameStatus

prop lockingattr state

meth init(G P)state:=still(G.nrows*G.ncols-G.nmines){P @state}

{Matrix.forAll G.fieldproc {$ R}

threadcase R of safe(K) then

{Wait K}lock

case @state of still(1) thenstate:=won {P @state}

[] still(N) thenstate:=still(N-1) {P @state}

else skip endend

[] mine(marked) thenskip

[] mine(exploded) thenlock

if @state\=lost thenstate:=lost {P @state}

endend

endend

end }end

end

Figure 12.12: The class GameStatus which implements a game monitor


DraftJune 10, 2002


creates a critical section. It prevents two threads from entering in critical sectionsat the same time. Such a lock for an object is provided by the declaration prop

locking in its class. The state transitions, with their corresponding call to P,are done in critical sections. This guarantees their execution in sequential order.

12.4.4 The user interface

We now build the user interface of Figure 12.7. A window has four parts: aleft panel with menu buttons and number entries, a label with a message, a gridof buttons (the game field) and check buttons controlling the digital assistant.The central element is the game field. Its buttons correspond to the squares ofthe Gameobject we are currently playing with. And its dimensions are those ofthe mine field, which could change for each new game! So we should be able toreplace the game field by another one, that is derived from the field of a newGameobject.

Our implementation of the user interface is presented in Figure 12.13. Awindow object Win is constructed from its high-level description Desc by means ofthe function Build . The latter function also create a handle for each componentand binds it to the handle field of the component’s description. Those handlesare objects that grant control over the corresponding widgets.

All the window components should be easily recognizable in Desc . The wid-gets td and lr are containers that place given widgets vertically (from top tobottom) and horizontally (from left to right), respectively. The buttons “Quit”and “New!” are associated with actions toplevel#close , which means to closethe window, and NewGame, the procedure that creates a new game and updatesthe interface. The game field is declared as a placeholder , i.e., a rectangulararea in which you can place a widget. And this is what we need: each new gamewill put its own game field into the placeholder.

The application is launched by first calling NewGame, which places a gamefield in the placeholder. It then makes the window visible with method show,whose parameter wait indicates that the method invocation blocks until thewindow is closed. When the latter happens, the application is terminated bycalling Application.exit .

The NewGameprocedure

The job of NewGameis to create a Gameobject, along with a GameStatus object,and to connect them to the user interface. The connection consists in changingsome widgets so that they refer to the newly created game objects. The mainchange is the game field, where each button correspond to a square in the game.The button’s appearance reflects the knowledge of the square and clicking on ituncovers the square.

The implementation of NewGameis presented in Figure 12.14. The proceduredefines:


DraftJune 10, 2002


% The window and its componentsNRHandle NCHandle NMHandleMessageHandle GridHandleAutoHandle ZeroHandle DeduHandleDesc=lr(title:"Minesweeper"

td(glue:nwbutton(text:"Quit" glue:nwe action:toplevel#close)label(text:"Rows:" glue:sw)numberentry(handle:NRHandle init:10 max:100 width:5)label(text:"Columns:" glue:sw)numberentry(handle:NCHandle init:10 max:100 width:5)label(text:"Mines:" glue:sw)numberentry(handle:NMHandle init:20 max:10000 width:5)button(text:"New!" glue:swe action:NewGame))

tdline(glue:ns)td(glue:nswe

label(handle:MessageHandle glue:we)placeholder(handle:GridHandle glue:nswe)lr(glue:we

checkbutton(text:"Autoplay" handle:AutoHandle)tdline(glue:ns)td(checkbutton(glue:w text:"Zero propagation"

handle:ZeroHandle)checkbutton(glue:w text:"Constraints"

handle:DeduHandle)))))Win={Build Desc}

% Play a new gameproc {NewGame}

...end

% Start the application!{NewGame}{Win show(wait: true )}{Application.exit 0}

Figure 12.13: Implementation of the graphical user interface


DraftJune 10, 2002


% Shows a message about the state of a gameproc {SetMessage S}

{MessageHandlecase S of still(N) then

set(bg:white text:"Still "#N#" squares to uncover")[] won then

set(bg:green text:"You WIN!")[] lost then

set(bg:red text:"You LOSE!")end }

end

% Play a new gameproc {NewGame}

NR={NRHandle get($)}NC={NCHandle get($)}NM={NMHandle get($)}G={New Game init(NR NC NM)}GS={New GameStatus init(G SetMessage)}

fun {MakeSquare I J R} ... end

ButtonMatrix={Matrix.mapInd G.field MakeSquare}Grid=

{Record.adjoin{Record.map ButtonMatrix

fun {$ Row}{Record.adjoin Row lr(glue:nswe)} end }

td(glue:nswe)}in

{GridHandle set(Grid)}end

Figure 12.14: Creation of a new game


DraftJune 10, 2002


NR, NC, NM are the parameters for the new game. Their value comes from thenumber entries in the window.

G is a new Gameobject (with NRrows, NCcolumns and NMmines).

GS is a new GameStatus object, monitoring G and calling procedure SetMes-

sage . The latter updates the message label by invoking MessageHandle

with the method set , whose form is chosen according to the state given inparameter.

MakeSquare is a function returning the description of the button correspondingto the square at row I and column J , whose status is known as R. It isdescribed below.

ButtonMatrix is a matrix of buttons. It is constructed from G.field , everysquare being mapped onto a button by MakeSquare .

Grid is the description of the game field. It is derived from ButtonMatrix , bytransforming the matrix into a combination of td and lr widgets.

The body of NewGamesimply put the new game field into the placeholder.

The function MakeSquare is shown in Figure 12.15. At the end of its body,we can see that the returned value is the description of a button with handle B.A thread binds actions to the first two mouse buttons, and changes the button’ssettings according to the value of R, which NewGamebinds to G.field.I.J . Thethread will first block until B is bound to a handle. It then waits for the value ofR, which is bound either by the Gameobject (when playing the square), or by aninference object.

Making it standalone

The functor that defines the standalone application is shown in Figure 12.16.It imports modules Application (to quit the application), OS (for randomlygenerated numbers), QTk (for the graphics) and Matrix (for the manipulation ofmatrices). The body of the functor brings together the elements of code that wepresent in the text.

12.4.5 The digital assistant

This section presents the various inference modules that make up the digitalassistant. We first determine their minimal interface. The modules Autoplay

and ZeroPropagator are given in full details. For the module Deduction , weonly give the ideas and the specification of the class. We then show how toconnect those modules to the application.

The inference modules are implemented as active objects. These modules areinstalled by the creation of objects of their class. A class C of such objects isrequired to provide the following methods:


DraftJune 10, 2002


fun {MakeSquare I J R}Bproc {PlayIt}

try {G play(I J)} catch _ then skip endendproc {MarkIt}

case {B get(text:$)} of "" then {B set(text:"?")}[] "?" then {B set(text:"")}else skip end

endin

thread{B bind(event:"<1>" action:PlayIt)}{B bind(event:"<2>" action:MarkIt)}

case R of safe(N) then{B set(text:"-")} {Wait N}% The square is safe & played, show result{B set(text:N relief:flat state:disabled

disabledforeground:blue)}[] mine(exploded) then

% The square has been played{B set(text:"X" relief:flat state:disabled

bg:red)}[] mine(marked) then

% The square has not been played yet{B set(text:"X" state:disabled)}

endend

button(handle:B text:"" width:2 glue:nswedisabledforeground:black)

end

Figure 12.15: Making a single square

functorimport Application OS(rand srand) QTk Matrixdefine

... % Definition of the classes Game and GameStatus

... % Definition of the classes for inference modules

... % Definition of the user interfaceend

Figure 12.16: The application functor


DraftJune 10, 2002


% A Switchable object can be set on/off at any timeclass Switchable

attr trigger: unit

meth set(B <= true )if B then @trigger= unit else unit =trigger:=_ end

endend

Figure 12.17: The Switchable class

Infer={New C init(G)} creates an inference object of class C for the game G.Its action depends on the knowledge of the game field. In other words, theexecution of the object is driven by dataflow on G.field .

{Infer set(B)} makes the object active (resp. inactive) if B is true (resp.false ). The inactivity means that the object is like suspended when seenfrom outside.

We prefer the method set to a method that just toggles the activity on/off. Thefirst one allows us to set the activity of an object according to the state of a checkbutton, for instance. The second method is less direct.

The class Switchable

We first factor out the activity switching mechanism in the class Switchable ,whose implementation is shown in Figure 12.17. One must inherit from Switch-

able before using it. The idea is that a Switchable object should be active ifand only if the value of its attribute trigger is determined. This is done simplyby performing

{Wait @trigger}

before doing the action. Making the object active is done by binding the currentvalue of trigger to unit . Making it inactive is done by

unit =trigger:=_

which changes trigger to an unbound variable and binds its old value to unit ,thereby avoiding a possible deadlock.

The classes Autoplay and ZeroPropagator

An Autoplay object for a game G is a Switchable object that will automaticallyplay a square when it is known to be safe. Its class, given in Figure 12.18, inheritsfrom Switchable . The method init creates, for each element R of G.field , athread that blocks until it can decide whether R matches the pattern safe(_) .In which case it waits for the trigger and plays the corresponding square.


DraftJune 10, 2002


% An Autoplay object will play square (I,J) if it is knownsafe

class Autoplay from Switchablemeth init(G)

{Matrix.forAllInd G.fieldproc {$ I J R}

threadcase R of safe(_) then

{Wait @trigger} {G play(I J)}else skip end

endend }

endend

Figure 12.18: The implementation of an automatic playing agent

A ZeroPropagator object for a game G implements the following inferencerule: for each element R of G.field , if R is safe(0) , then all the elementsaround it are known safe. The class is presented in Figure 12.19. This classgives the basic inference mechanism which is present in most implementationsof the game. Our originality lies in the independence of our object from thegame’s implementation. Others often completely integrate the mechanism intothe game’s implementation.

The advanced inference class Deduction

The goal of this module is to infer “as much information as possible”. We relyon the fact that the knowledge about the mine field is a partial description ofthe solutions of a problem. From that point of view, the class Deduction isconceived as a problem solver, whose problem is described below.

We state the minesweeper problem with respect to a mathematical model ofthe game. Our model is largely inspired from the matrix BinField in the classGame. The hidden mine field is a binary matrix M whose elements equal to 1represent the mines. So knowing that the square at (i, j) is safe or mined is statedas

Mij =

{0 if the square is safe1 if the square is mined

(12.1)

and knowing that it has k mines in its neighborhood is equivalent to∑ {Mi′j′

∣∣∣ |i′ − i| ≤ 1 and |j′ − j| ≤ 1}

= k. (12.2)

The problem consists in determining every possible Mij , having several linearequations over integers like (12.1) and (12.2).


DraftJune 10, 2002


% If (I,J) is safe(0), then bind all neighbors to safe(_)class ZeroPropagator from Switchable

meth init(G){Matrix.forAllInd G.field

proc {$ I J R}thread

if R==safe(0) then{Wait @trigger}for X in {Matrix.neighborList G.field I J}do

X=safe(_)end

endend

end }end

end

Figure 12.19: The implementation of an agent propagating knowledge

The class Deduction implements the required methods for inference classes,plus one extra method:

{Infer search(border(I J))} performs a local search around the square at(I , J). The goal is to find the values that are common to all the solutionsof the minesweeper problem. The search explores the values of the squareson the same border of the frontier between known and unknown squares asthe square at (I , J).

The implementation of the class uses computation spaces.

Adding the inference modules to the application

Thanks to the application’s architecture, only the procedure NewGameneeds slightmodifications. We first create the inference objects and action procedures byadding the following lines in the declaration part:

Auto={New Autoplay init(G)}Zero={New ZeroPropagator init(G)}Dedu={New Deduction init(G)}proc {AutoAction} {Auto set({AutoHandle get($)})} endproc {ZeroAction} {Zero set({ZeroHandle get($)})} endproc {DeduAction} {Dedu set({DeduHandle get($)})} end

We then update the widgets controlling the activity of those objects by addingthe statements

{AutoAction} {AutoHandle set(action:AutoAction)}{ZeroAction} {ZeroHandle set(action:ZeroAction)}


DraftJune 10, 2002

12.5 Exercises 735

{DeduAction} {DeduHandle set(action:DeduAction)}

at the end of the body of NewGame. It remains to bind the search tool of theobject Dedu to each square. We add the following procedure in the declarationof MakeSquare :

proc {SearchIt}if {DeduHandle get($)} then

% Launch the search{Dedu search(border(I J))}

endend

Note that the call to {DeduHandle get($)} returns true only if the Dedu

object if active. That procedure is bound to an event by adding the followingstatement

{B bind(event:"<3>" action:SearchIt)}

in the body of MakeSquare .

12.5 Exercises

1. Cryptarithmetic. Write a program to solve all puzzles of the form “Word1plus Word2 equals Word3”. The words should be input interactively. Usethe solution to the SEND-MORE-MONEY problem given in Section 12.2.1as a guide. The user should be able to stop the search process if it is takingtoo long. Use the Search.object tool of Chapter 8.


DraftJune 10, 2002



Documents

Draft June 10, 2002 688 Distributed Programming (incomplete) might have already edited the object in question, leading to a conﬂict. There are many ways to resolve conﬂicts. F