IntelligentPad as meme media and its application to multimedia databases

Information and Software Technology 38 (1996) 201-211

IntelligentPad as meme media and its application to multimedia databases Yuzuru Tanaka

Division of Electronics and Information Engineering, Hokkaido University, N-13, W-8, Sapporo, 060, Japan

Abstract

Computers as meta media are now evolving from augmentation media vehicles to meme media vehicles. While an augmentation media system provides individuals, groups or organizations with a seamlessly integrated environment of various tools and documents, meme media system provides communities or societies with further functions to edit, to distribute, and to share tools and documents. Meme media work as memes, i.e. cultural genes. Documents and tools on meme media can easily replicate themselves, recombine themselves and are naturally selected by their environment, namely the society of their authors and users. Their accumulation in their users’ community will form a meme pool, which will bring rapid evolution of documents and tools. The IntelligentPad architecture provides a standard framework called a pad that works as a meme medium. When applied to multimedia databases, it will allow us to manage not only articulated objects but also non-articulated objects in a generic manner.

Keywords; Media architecture; Object-oriented system; Multimedia database

1. Introduction

Today’s personal computers are considered as augmentation media that provide us with various tools to augment our individual capabilities. Some of them are expanding their target of augmentation from individuals to groups, or further to organizations. They are called respectively group- wares and enterprise integration systems. The IntelligentPad system developed at Hokkaido University since 1987 further expands the target of augmentation from groups and organizations to communities or to societies. While people in a group or in an organization share a definite common goal such as making a decision, designing a system, or solving a problem, people in a society share their achievements and reuse them to produce new achievements. They share a pile of their achievements, namely their culture. This paper reviews the IntelligentPad system from the viewpoints of its motivation, its design philosophy, its system architecture, its applications and its current state of the art.

Media for the externalization of people’s knowledge enable us to record and archive them extrasomatically, and to distribute them in their community. This encourages people to reuse the archived knowledge fragments to edit new ones, which are again accumulated upon the shared pile of knowledge. This pile is what we call a culture. Biological evolutions are based on genes. We require similar genetic media for the augmentation of societies to promote their cultural evolution. Such media with their contents should be able to replicate themselves, to recombine themselves, and to be naturally selected by their environment. They may be called meme media since they carry what

email:[email protected]

0950-5849/96/$15.00 0 1996 Elsevier Science B.V. All rights resewed SSDI 0950-5849(95)01074-2

Dawkins [ l] called ‘memes’. Their environment is the society of their producers and consumers, namely, authors and users. Stefik [2] also pointed out in 1986 the impor- tance of understanding and developing an interactive knowledge medium that embodies the characteristics of memes. The accumulation of memes in a society will form a meme pool, which will work as a gene pool to bring a rapid evolution of knowledge resources shared by this society. This explosive increase of knowledge resources forms a sufficiently large meme pool to further increase the chance and the variety of their recombination.

It is noteworthy that the two big pioneers of hypermedia systems respectively focused on these two types of media. Engelbart [ 3,4] mainly focused on augmentation media in NLS and Augment. On the other hand, the worldwide publishing repository that Nelson [ 51 aimed at in Xanadu is nothing but a meme pool. Hypermedia systems as augmentation media are included in a lot of systems and proposals [6-lo]. However, they have not sufficiently developed themselves to augment societies and their cultures. Hypermedia systems as meme media, on the other hand, have been focusing on documents and not on tools even in their most recent versions like WWW and Mosaic [ 11,121 . These two types of hypermedia system technologies have not yet been well integrated.

While personal computers have dramatically simplified the editing of multimedia documents, they cannot yet allow us to easily edit existing tools to create new tools. The editing of documents and tools by end-users is fundamental to the evolution of their meme pool. The IntelligentPad system architecture uniformly represents both documents and tools as pads, i.e. reactive media objects that look like paper sheets on the display screen [ 13-161 Pads work as

202 Y. Tanakallnformation and Software Technology 38 (1996) 201-211

meme media for the editing, the replication and the natural selection of various types of documents and tools. A pad is a persistent object. Pads can represent multimedia documents, even including videos and sounds, server programs such as database servers and various network servers, and also various application programs. Their environment is also represented as a pad on which any pads can be arranged. In an IntelligentPad system, you can easily compose any document or any tool by directly pasting some pads on top of another. Such a paste operation simultaneously defines both the layout of its components in the composed pad and the functional linkage among component pads. Composite pads are also simply referred to as pads. Pads can easily be replicated. Copies of the same pad inherit the same functional mechanism of the original, but change their state independently from each other. They can easily be edited by pasting some other pads on themselves, or by peeling their component pads off themselves. Besides, they can easily be transported from one system to another via networks or off-line media.

The transportability of pads, however, is not sufficient for the distribution and exchange of pads. It alone will not form a meme pool. We need a marketplace where we can show, browse, sell, buy, and exchange various pads. Some off-line distribution media will be able to use conventional market systems such as book stores and catalog shopping services to form such a marketplace. We have already developed a worldwide open market system of pads. This system uses WWW and Mosaic to provide a worldwide pad repository and a hypermedia catalog of pads for the navigation of this repository. Once you find a desired pad, you can import this pad into your IntelligentPad system just by directly clicking an anchor button. This market is open to end-users so that they can freely open their own pad shop, and navigate through a worldwide network of pad shops just for browsing or for seeking specific pads.

2. Meme media as decomposable and reactive compound documents

Meme media are such augmentation media that can carry any kinds of knowledge resources and dynamically inter- relate them to cooperate with each other. These knowledge resources include multimedia documents, basic tools, application tools, knowledge and their inference tools based on various knowledge representations, and interface tools to control and utilize various external devices and systems. Meme media should provide these knowledge resources as generic components so that their users can easily combine them to edit composite documents and tools. Meme media should also allow us to distribute them as decomposable objects. One possible solution to satisfy these requirements is the adoption of the decomposable and reactive compound- document model.

The adoption of the compound-document model began in

the early 1980s with DTP (Desk Top Publishing) [ 171. DTP enabled us to layout various types of media on a single page. The WYSIWYG design showed the print image on the display screen. Around the mid 198Os, compound documents were further extended to allow the embedding of video clips as their components. Then the object-oriented system design allowed us to embed programs in compound documents. Such documents became reactive compound documents. Around the mid 1980s the synthetic paradigm of the object-oriented programming developed various GUI (Graphical User Interface) toolkit systems [ 18,191, and by 1986 this trend encouraged our group to apply the toolkit approach to the design of reactive compound documents. We retained the decomposability of compound documents for their reuses. This led to our proposal of the decomposable and reactive compound-document model and resulted in the development of the IntelligentPad system. Pads are inherently different from widgets. While each pad always consists of a display object and a model object, widgets might be considered as display objects without a model. The display object of a pad consists of a view and a controller which respectively define its appearance on the display screen and its reaction to user events, while its model object defines its internal state and behaviour.

3. IntelligentPad as meme media

3.1. From object-oriented architectures to media-based architectures

Current computers treat various types of intellectual resources. Unless they can provide a dedicated functional linkage between any pair of these different types of resources, they fail to integrate these resources (Fig. l(a)). By a functional linkage is meant not only a data communication linkage but also an application linkage through message exchanges. Different pairs require different types of linkages. If there are n different types, we require 0(n2) different types of linkages. This is the essential difficulties of open integration systems that are open to the future addition of new intellectual resource types.

The IntelligentPad architecture has solved this problem by separating container media from their contents, and standardizing the logical structure and the interface of the container media. Each primitive pad consists of its shell and its functional contents. Its shell defines its standard media structure and interface. It is up to the developer of each pad how its functional contents are implemented in the standard shell. While the IntelligentPad is based on an object-oriented architecture, it further restricts its architecture. There- fore, we call such architectures media-based architectures (Fig. 1).

In a media-based architecture, only one type of functional linkages is used to connect any pair of shells. In the IntelligentPad, each shell has an arbitrary number of jacks

Y. Tanakallnformation and Software Technology 38 (1996) 201-211 203

P 4 ~24

(a) Object-oriented architectures require different linkage types for different pairs of objects.

(b) Media-based architectures require only a single standard linkage type to connect two standard container media.

Fig. 1, Object-oriented architectures and media-based architectures.

called slots (Fig. 2). It also has a single pin-plug to connect itself to one of the slots of another shell. The shell architecture and the standard linkage facility are provided by the kernel of the IntelligentPad systems. Neither users nor pad developers have to worry about them. A shell with its contents is called a pad.

3.2. IntelligentPadS basic architecture

A pad in the IntelligentPad consists of a display object and a model object (Fig. 3). Its display object defines both its view on the display screen and its reaction to user events, while its model object defines its internal state and

. . . . ,, _. . . GsPltT.

a pad as a media object

connection jacks (slot list)

w pin plug

Fig. 2. The logical structure of a pad that allows its generic definition.

Controller View message- sending

Model

message- sending update propagation

Fig. 3. The internal structure of each pad.

behaviour. Wide varieties of documents and tools can be

represented as pads. You may use paste operations in arbitrary ways, for example, to overlay multiple translucent pads of the same size, or to arrange multiple pads on the same base pad. Users can easily replicate any pads, paste pads on another, and peel a pad off a composite pad. These operations can be equally applied to both any primitive pads and any composite pads. When a pad P, is pasted on another pad P, , the pad P, becomes a subpad of P,, and P,

becomes the master pad of P2. No pad may have more than one master pad (Fig. 4). Pads are decomposable persistent objects. You can easily decompose any composite pad by simply peeling off any primitive or composite component pad from its master pad.

Fig. 5 shows a bookshelf and books that are all con- structed by pasting various primitive pads. The opened book defined as a composite pad shows a text with a scroll bar on the left page and a map of Japan with several bar meters on the right page. This scroll bar pad and these bar meter pads actually share the same function that detects the mouse location and sends the underlying pad a value between 0 and 1 depending on the detected relative location. Therefore, you may replace the scroll bar with a copy of these bar meters. When you paste a bar meter on the text pad, you have to connect this bar meter to the text- scrolling function of the underlying text pad. You can specify this connection just by selecting the scroll slot from the list of slots defined by the text pad.

The application-linkage interface of each pad is defined as a list of slots. Each slot can be accessed either by a ‘set’ message ‘set (sZot__name) value’ or by a ‘gimme’ message ‘gimme (slot-name)‘. Each of these two messages invokes the respective procedure attached to the slot. Each slot s, may have two attached procedures, proqset for the ‘set’ message and pr~c~,~~,,,,,,~ for the ‘gimme’ message. The default for p~oc,,,,~ stores the parameter value into the slot register, while the default for ~TOC;,~~,,,,,,~ returns the slot register value. Its slots and attached procedures define the internal mechanism of each pad. They are defined by its developer.

When a pad P2 is pasted on another pad P,, the Intel- 1igentPad constructs a linkage between their view parts (Fig. 6). This defines a dependency from P, to PI. If P, has more than one slot, we have to select one of them to associate it with P2. This selection can be specified on a connection sheet. The selected slot name is stored in a standard slot of the subpad P2 named connectslot. A subpad can send either ‘set I connectslot(value) ’ message

204 Y. Tanaka1 Information and Software Technology 38 (1996) 201-211

Pl no p2 p3

(a) a composite pad

(b) application linkage among the three pads

Fig. 4. The application linkage in media-based architecture.

Fig. 5. Composite pads define a bookshelf and books.

p2

or ‘gimme 1 connectdot’ message to its master, while the master pad, when its state is changed, can send some of its subpads an ‘update’ message without any parameter to

5_ set [ < slot_name > I < value > propagate an update event. The interpretation of the update message again depends on the implementation of the sender and the receiver pads. It is usually used to inform the

Pl subpads of a master’s state change. The up arrow before connectslot means that the slot name stored in connectslot

Fig. 6. The standard message interface between pads. becomes the real parameter. In the definition of pads,

Y. Tanakallnfonnation and Sofrware Technology 38 (1996) 201-211 205

programmers use ‘pset (value) ’ and ‘pgimme’ messages, which the IntelligentPad kernel automatically converts to ‘set t connectslot (value)’ and ‘gimme t connectslot’ messages. IntelligentPad allows us to disable some of the three standard messages, ‘set’, ‘gimme’ and ‘update’, which can be specified on the connection sheet. The two messages ‘set s v’ and ‘gimme s’ sent to a pad P are forwarded to its master pad if P does not have the slot s.

Besides these three standard messages, any pad can send some additional standard messages for geometrical operations to its master as well as to its subpads. Among them are move, copy, delete, hide, show, open, close, resize, and paste messages. These are the messages that can be applied to any pads without specifying any slot connection. The set of these standard messages defines a standard interface between pads.

3.3. IntelligentPad as a meta tool

Any pads accept copy and shared-copy requests. Shared copies of the same pad share the same state, i.e. the same model object. The state of a composite pad is defined as the state of the base pad. Shared copies, however, cannot share a user event applied to one of them unless it only changes the shared state. A user event may change the pad view without changing its state. Hence, we require an event sharing mechanism as an independent primitive function. In the IntelligentPad architecture, every independent function should be implemented as an independent pad so that its generic function may be applicable to any pads. A special pad called a FieldPad represents the field of user events. You may put any composite pads on a FieldPad. Shared copies of this composite pad share every user event applied to it.

Pads can be transported through networks to any other IntelligentPad system. Pad transportation tools are also provided as pads. Shared copies distributed over a network can still keep sharing the same state.

3.4. Proxy pads and database applications

Fig. 7(a) shows a form interface to a relational database. The IntelligentPad works as a form construction kit for relational databases and object-oriented databases. The base pad in this figure works as an interface to the database. Such a pad that works as a proxy of some external object is called a proxy pad. Among such external objects are industrial plants, databases, computer-controlled devices, and numerical computation programs running on super computers. A proxy pad for some external object provides the following. First, it provides an interface program to communicate with this external object. Second, it provides a logical pad structure, namely a list of appropriate slots. Third, it provides a physical pad view on the display screen.

A database proxy pad looks like a blank sheet, and provides a list of slots including the query slot, the result

(a) a form interface to an automobile database

(b) a form interface with a photo slot storing a composite pad

Fig. 7. A form interface to a relational database.

slot, the current-record slot, the search slot, the insert slot, the delete slot, the next-candidate slot, and the previous- candidate slot. The query slot holds an SQL or an object- oriented SQL query. When the proxy pad receives ‘true’ in its search slot, it sends the SQL query stored in the query slot to the database system to perform this search. When a search result is sent back from the database, the proxy pad stores this list of records in its result slot. At the same time, it makes its internal variable ‘cursor’ point to the first record in the record list, and stores this record in the current-record slot. When the proxy pad receives ‘true’ in its next-candidate slot or in its previous-candidate slot, it respectively changes its variable ‘cursor’ to point to the next or to the previous record, and replace the record in the current-record slot with this new record.

You may easily paste various pads on a database proxy pad to define a visual interface of its database. The RecordPad is also a blank sheet. It reads out a record-type value from its master pad and stores this record in its primary slot. It allows us to add an arbitrary number of special slots called attribute slots. Each attribute slot, when requested to send back its value, reads out the record in the primary slot and gets the value of the attribute having the same name as this

206 Y. Tanaka1 Information and Software Technology 38 (1996) 201-211

attribute-slot name. If the record does not have the same attribute name, this attribute slot returns the value ‘nil’. When pasted on the database proxy pad with its connection to the current-record slot of this proxy pad, the RecordPad works as a base pad to define a form representation of each record that will be retrieved from the database proxy pad. Different from GUI forms, this form with a RecordPad as

its base pad can be easily copied with its content record, and can be sent to other users or some tool pads.

In its typical use on a database proxy pad, the RecordPad divides each retrieved record into a set of attribute-value pairs. Each attribute value is set to the slot with the same name as its attribute name. Depending on the value type of each attribute slot, you may connect a text pad, an image pad, a drawing pad, or a video pad to this slot. You may arbitrarily design the layout of these viewer pads on the RecordPad. A database proxy pad with a RecordPad pasted with some viewer pads is called a form-based DB viewer.

registered query. The retrieved result will be shown on this form-based DB viewer. If you want to show users of the automobile database in Fig. 7(a) various options on several items such as the body colours and the wheel cover, you can just paste two query issuing pads on the photo pad and store this composite pad in the photo attribute of the database. A mouse-click on each of these buttons will pop up the corresponding form-based DB viewers showing the related possible choices. Fig. 7(b) shows such a photo pad and a popped up form showing three possible choices of the wheel cover.

4. Multimedia database applications

4.1. Multimedia objects and their articulation

For the setting of a query in the query slot of a database proxy pad, you can simply use a text-input pad. The text- input pad is pasted on the proxy pad, and connected to the query slot of the proxy pad. An SQL query written on this text-input pad is sent to the query slot of the proxy pad. Instead of using such a simple text-input pad, you may use any composite pad to generate an SQL query.

The IntelligentPad can save and load its pads to and from files using their save-format-string representations. The save-format-string representation of a composite pad consists of a list of pad IDS (Identifiers) of its component pads, their geometrical relations in this composition, and their states including their current slot values. It, however, includes no programs. The save-format-string representation of any pad can also be stored as an attribute value into a database. This implies that we may consider relations with such attributes whose values are composite pads. Such an attribute is called a pad-type attribute. Suppose that a RecordPad pasted on a database proxy pad has a pad-type attribute slot. To view its values in the form of composite pads, we can paste a SaverLoaderPad on this RecordPad and connect this pad to this pad-type attribute slot. When it receives a save- format-string from its master pad, the SaverLoaderPad reconstructs the corresponding composite pad. When a composite pad is pasted on it, the SaverLoader pad sends the corresponding save-format string to its master pad.

In multimedia systems we have to deal with various types of objects. They should be first classified into two categories, i.e. container objects and content objects. Container objects are container media that carry content information. Books, pages, cards, display windows and communication packets are all examples of container objects. They provide the respective structures and operations as those of these containers. A media object is defined as a container object with its content objects. Content objects are further classified into two categories. Some of them are clearly articulated, i.e. easily machine recognizable from their representations, while others are not. Machine readable texts have such content objects as characters, words, phrases, sentences, paragraphs, sections, and chapters. They are all articulated

objects. Tables have such content objects as entities, columns and rows. Tabulation tools clearly articulate these objects. Each table format itself is a container object. Charts have such content objects as items, item values, and curves. A curve here is considered as a sequence of item value objects. Chart tools articulate these objects. Each chart type itself is a container object. Figures drawn by drawing tools have such content objects as points, line segments, shapes, and their composites. Drawing tools articulate these objects. The ground of each figure is a container object. All the content objects above are articulated, while most content objects in images, movies and sounds are not articulated.

Instead of just storing a photograph of a car in the photo A photograph of a town may show people, roads, cars, attribute of an automobile relation, you may store a com- and buildings. Images have such non-articulated content posite pad with a photograph pad as its base and several objects. They are not easily machine recognizable. Movies query-issuing button pads on this base. Fig. 7(b) shows have both articulated and non-articulated objects. Frames this extension. Its RecordPad has a photograph pad pasted are articulated container objects, while cuts are either with two query-issuing button pads. Each query issuing articulated or non-articulated content objects depending on button pad is a priori associated with a form-based DB whether or not the cut change signal is available. Physical viewer. This DB viewer has an a priori registered SQL objects shown in movies are all non-articulated content query in its query slot. This query may be issued either to objects. As to sounds and voices, we have to distinguish the same database or to a different database. When mouse- coded ones from recorded ones. Coded sounds have such clicked, a query issuing button pad pops up the registered articulated content objects as notes, bars and instruments. form-based DB viewer and instructs this pad to issue its These content objects are hard to recognize in recorded

Y. TanakalInformation and Sofhyare Technology 38 (19%) 201-211 207

music. However, we often want to identify a priori specified time segments of music or speech. They are inherently non-articulated content objects, but the a priori given time frames to identify them are themselves articulated objects.

Our main concern here is how to articulate non-articulated multimedia objects. Multimedia systems should allow us to select directly and to manipulate directly not only articulated objects but also non-articulated content objects. The most widely used general solution is the use of a reference frame object for each content object. For recorded music or a speech, it defines the shortest time segment that includes one of the music or speech portions you want to identify. This reference frame object indirectly specifies the corresponding music or speech portion. For an image, it defines the minimum rectangular area that covers one of the content objects you want to identify in this image. This reference frame object indirectly specifies the corresponding area of the image. Reference frame objects are articulated objects. Time segments work as temporal reference frames, while rectangular areas work as spatial reference frames.

The reference frame object for a cut in a movie defines a time segment. For an object appearing in a movie, its reference frame object defines a mobile variable-size rectangular area that minimally covers this object in every video frame showing this object.

Pads in IntelligentPad are suitable for the representation of the container objects, media objects, and reference frames in multimedia systems. Pads represent multimedia containers as windows or window widgets. They can easily be combined graphically to represent books, pages, cards, and also compound document frameworks. Since pads can hold any information and can provide slots to access their contents in various ways, they can easily represent media objects with sufficient interface to access their content objects. For the access of non-articulated content objects in a media object, we can provide this media object with a special slot named ‘reference-frame’ that receives the location and size of a reference frame and returns the corresponding portion of its content information. A ‘set’ message with the location and size of a reference frame as its parameter value is used to send this parameter to the reference-frame slot. After this operation, a gimme message is used to read the corresponding portion of the content information.

Spatial reference frames can be represented as transparent pads that cover the target content objects. They can be pasted on top of their target media object pads to minimally cover their target content objects. These reference frame pads provide slots as necessary interface to the target contents. Among them are the name slot to hold the name of the content object, the location slot to hold its relative location on its master pad, and the extent slot to hold its size. The oid (object identifier) of this reference frame pad semantic- ally works as the oid of the corresponding content object. Any pads with the above-mentioned functionality can work as reference frame pads. IntelligentPad, however, provides

some standard pads for the usage as reference frame pads. Among them are anchor pads and viewer pads. An anchor pad allows us to register a reference pointer to any composite pad. When mouse-clicked, it loads this registered

pad and pops it up on the screen. Anchor pads without any further functionality are not connected to any of the slots provided by its master media object pads. A viewer pad, on the other hand, is connected to the reference-frame slot of the media object pads. It has the contents slot to hold the copy of the corresponding portion of the content information. It also works as an anchor pad.

4.2. Operations and relations over multimedia objects

Articulated objects are units of operations. Among operations in multimedia systems are:

(1) (2) (3)

File operations (save, load). Edit operations (create, delete, edit). Quantification, i.e. condition specification in database queries.

(4) Link operations.

(5) Context specification.

(6) Composition operations.

(7) Overlay operations.

Edit operations are applied to each single object, while composition operations are applied to a set of multiple objects to combine them and to obtain a compound object. Each object may be referred to in specifying some of its properties. Each object may work as an anchor. It may link itself to another object, or work as a destination anchor that is pointed to by another object. The specification of an object also specifies its context that is also an object. For example, we can specify those figures showing a square as one of its components. Here we only specify a shape object. Those figures are specified as the contexts of such objects. Multimedia documents and components can be overlaid in multiple layers using the a-channel technology.

These operations are applicable not only to articulated objects, but also to reference frames of non-articulated objects. Original images, movies and sounds are kept unchanged in files. File operations save and load only reference frame objects. Edit operations are applied only to the copies of the portions selected by reference frame objects. These copies become properties of the reference frame objects. Quantification is also applied to reference frame objects. It specifies only those properties that can be easily calculated in real-time. Otherwise, such quantification cannot be processed within a reasonable amount of time. In images, spatial relationships between two reference frame objects are such properties. In movies, we can treat spatio- temporal relationships among reference frame objects. Each reference frame object can work as an anchor. It can link its corresponding non-articulated object to another object, or it works as a destination anchor pointed to by another object. Each reference frame object can tell which

208 Y. Tanaka IInformation and Software Technology 38 (I 996) 201-21 I

object is its context. Reference frame objects can be combined to a compound object, and also they can be overlaid with multimedia documents and components.

The pad representation of reference frame objects uses anchor pads for those content objects that do not require edit operations. For those requiring edit operations, it uses viewer pads.

4.3. Application linkage in multimedia

Application linkages in object-oriented systems are represented by either the object embedding or the object linking. The object embedding embeds a media object in another media object. It just defines a geometrical contain- ment relationship or further defines functional linkage between the two media objects. The object embedding can easily be defined in IntelligentPad by the paste operation. The pasted subpad may just sit on the master pad without any functional linkage, or it may be connected to one of the slots provided by the master pad to establish a functional connection. An extreme case of the object embedding in IntelligentPad is the pasting of one translucent pad on another pad of the same size. A translucent pad is a pad whose background colour is set translucent. IntelligentPad allows us to make any pad translucent. This extreme case is used to overlay multiple layers of information that are mutually geometrically related to. Computer mapping systems conventionally use such overlay representation.

The object linking uses the wiring technique to connect spatially separated two objects. Navigation links (cold, warm and hot links) in hypermedia and transclusion in Xanadu are examples of the object linking. In general, any messages can be exchanged between the two objects through the object linking. IntelligentPad uses the shared copy operation for the object linking: it also provides the wiring pad. Its shared copy works together with the original as two terminals connected by a cable. These two terminals with a cable transport any value from one to the other. It may work as either a bidirectional channel or a unidirectional channel. These two modes can be easily selected by the specification of its ‘mode’ slot. Anchor pads implement cold navigation links. Wiring pads implement warm and hot navigation links.

4.4. Multimedia databases and IntelligentPad

Multimedia databases have to deal with several different categories of objects. They are container objects, media objects, articulated content objects and reference frames for non-articulated content objects. IntelligentPad represents media objects as primitive pads and it represents container objects as primitive pad classes. Reference frames are represented as anchor pads or viewer pads.

Pads are all object instances. Primitive pads have their corresponding pad classes, while composite pads are defined by pasting primitive pads and have no corresponding

classes. Since object-oriented databases manage object instances based on their class structures, they can manage only primitive pads as objects, and not composite pads directly as objects. They can, however, manage the master- subpad relationship between pads as an object relationship. The save format string representation of a composite pad can be stored in databases as a string value.

Here we consider a multimedia database architecture based on an object-oriented database and IntelligentPad. For each primitive pad class, our database has a corresponding relation that has attributes corresponding to all the data slots of this pad class. This relation stores all the pads of this class, and it also has another attribute to store oids of pads. The master-subpad relation stores all the master- subpad pairs of oids with their subpad’s locations on the corresponding masters.

Since articulated content objects have their corresponding classes, they can be managed by databases. IntelligentPad, however, cannot represent them as pads: it treats them as slot values of primitive pads that represent the corresponding media objects.

4.5. Movie databases

In this section, we pick up movies to show how their objects are represented as pads and managed by a database. IntelligentPad represents each movie as a pad. This pad shows a Quicktime movie. It has a data slot that always indicates the current frame number. A reference frame for a physical object shown in a movie is a transparent pad that changes its size and location to cover minimally this same object in each frame of this movie pad. Users can specify its location and size in some sample frames using such a tool as shown in Fig. 8. This tool is itself defined as a composite pad. It enables you to specify the location and the size of each reference frame pad in arbitrarily selected

Fig. 8. A tool used to specify the location and size of each reference frame

pad in some sample frames of a movie.

Y. Tanakallnformation and Software Technology 38 (19%) 201-211 209

sample frames of an arbitrarily given movie pad. Each reference frame pad interpolates its location and size in

other frames from the given values in sample frames. Reference frame pads have the following data slots: object-name, frame-number, location, and size.

The database includes two relations, one for movie pads and another for reference frame pads:

(1) Movie (oid, title, author, date, location). (2) MovieReference (oid, object-name, movie_oid,

frame-#, location, size), where oid in MovieReference is the oid of the reference frame pad.

For simplicity, it is assumed that all the frames that show

the same object are recorded in the relation MovieReference. In a practical system, it is sufficient to store sample frames in MovieReference.

This rather simple system architecture provides large capabilities for the manipulation of movies. File operations are supported. Edit functions are also supported if we use viewer pads for reference frame objects.

Our system architecture allows varieties of quantification on various objects. Needless to say, for any quantification on articulated objects, it allows us to quantify non- articulated objects. Suppose that there are lots of movies showing butterflies and flowers, and that we want to find out all the movie cuts in which a swallowtail is flying over a dandelion. We assume that butterflies and flowers are all minimally covered with reference frame pads. This retrieval request can be described by the following SQL query:

select movie.oid, mrefl .oid, mrefl .frame_# from movie in Movie; mrefl , mret2 in MovieReference where movie.oid =mrefl .movie_oid=

mret2. movie_oid and mrefl . frame-# = mref2. frame-# and mrefl .object_name= ‘swallowtail’ and mret2. object-name = ‘dandelion’ and mref 1. location. y + mref 1. size. y <

mref2.location.y and mrefl.location.x+mrefl .size.x> mref2.1ocation.x and mret2.location.x+mref2.size.x>mrefl.location.x.

This result relation can be grouped by the movie.id and mrefl.oid to obtain two attributes movie.oid and mrefl. frame_#/(movie.oid, mrefl .oid). From the second set- value attribute, we can calculate the minimum min(mref1. frame_#/(movie.oid, mrefl.oid)) and the maximum mar(mref.frame_#/(movie.oid, mrefl .oid)) of this set. Let pad be a function from a pad id to its corresponding pad. Then the movie cuts we want are obtained as pad

(movie.oid)I Ctl.tzl, where tl = min(mref.frame_#/(movie. oid, mrefl .oid)), t2 = max(mref.frame_#/(movie.oid, mrefl .oid)), and 1 ,t,,tzl denotes the selection of the cut between these two frame numbers.

Our architecture allows us to span a link from any non-articulated object in any movie to another pad. An anchor pad is used as a reference frame to cover this non-

articulated object. The specification of a reference frame

also specifies its underlying movie pad and some of its

frame numbers. Therefore, context specification functions are also provided. Composition operations and overlay operations are both provided by IntelligentPad.

Our system architecture can also deal with images and sounds in a similar way.

4.6. Articulated objects in media objects

Our architecture uses its object-oriented database to manage articulated objects in media objects. Here we pick up texts to show how our architecture manages articulated objects in texts.

IntelligentPad represents each text as a text pad. A text pad is associated with a text file storing its text string. It works as a screen editor of its text file. Users can easily move its cursor, scroll the text, and specify any portion of the text, all through direct manipulations using a mouse. A text pad also allows its user to register hypertext link anchors and to invoke them by a mouse operation. It provides a special slot ‘edit’ to access any articulated object. Its access by a ‘set’ message with a parameter (ref (10~1)

(10~2)) makes this text pad update its current-object register to the text portion between two locations (10~1) and (10~2). A set message with (insert (lot) (text)) makes this pad insert the text string (text) at the location ( lot > . A set message with (de1 (lot 1) ( 10~2)) makes this pad delete the text portion between (lot 1) and (10~2). An access of this edit slot by a gimme message makes this pad return the value of its current-object register.

In our multimedia database architecture using Intelligent- Pad and an object oriented database, texts are stored not in text files, but in the database. Each text pad is therefore modified so that it can read and write the text contents to and from the database.

Such an architecture supports file operations, quantification, link operations, and context specification on articu-

lated content objects. Furthermore, since text pads can be pasted together with any pads, they allow composition operations and overlay operations.

Similarly, articulated content objects in figures and tables are all managed by our database, while figures and tables themselves are represented as pads that access the database to display and to edit their contents.

5. Pads as meme media and a worldwide meme market

We are also interested in the electronic publication of academic papers. They may include video clips and sounds. Furthermore, each formula in them is a pad with a program to calculate this formula. Every chart stores the raw data used to plot this chart. You can make a copy of a formula in such a reactive document and paste it on your own tool pad to evaluate this formula from your own viewpoint.

210 Y. Tanaka/Information and Software Technology 38 (19%) 201-211

Such electronic documents may further contain application tools as embedded document components.

The examples we mentioned above include a lot of useful tools. Some of them are composite pads, some are component composite pads, and others are primitive pads. They are equally useful for the users belonging to the same subcommunity. The distribution and the exchange of useful tools and documents in such subcommunity will surely stimulate the evolution of its culture.

Pads can easily be replicated and edited, and easily transported from one system to another via networks or off-line media. If the two systems share the same class library, the transport of a composite pad instance sends the destination system only the save-format-string representation of this composite pad. Otherwise, it further requires the transportation of some class definitions.

However, the transportability of pads is not sufficient for the distribution and exchange of pads. It will not form a meme pool. We need an open marketplace where each end- user can show, browse, sell, buy and exchange various pads.

Our open marketplace architecture uses WWW and Mosaic. While WWW works as a worldwide pad repository, Mosaic provides a hypermedia catalog of pads to navigate this repository. We assume that every site in the community of our concern instals an IntelligentPad system with the same class library. Each page of Mosaic describes various pads using texts and images (Fig. 9). Besides the ordinary functions of Mosaic, this catalog has the following extended functions. Pad names in textual descriptions and visual images of pads work as special anchors. A mouse- click on one of these special anchors will pop up a new IntelligentPad window with a copy of the requested pad. The original of this pad may be stored anywhere in the world. It is stored there using the save format string representation. When the anchor is clicked, Mosaic issues a file transfer request to the HTTP (Hypertext Transfer Protocol) server at this remote site. After a while, the local Mosaic receives this file, and then invokes the IntelligentPad

Fig. 9. A pad catalog on Mosaic and an IntelligentPad system with a

window including the imported pad.

system to reconstruct the pad using its save format string representation. This reconstruction does not differ from the pad reconstruction process necessary for the loading of a pad from a local file.

This mechanism is quite easily implemented using the Mosaic’s facility of mapping a customer-defined file-name- extension to an invocation of an application program. We used ‘PAD’ as the customer-defined file-name-extension, and mapped this to the invocation of a special file loader in the IntelligentPad system. While the implementation mechanism is quite simple, it has opened a new vista in the distribution and exchange of knowledge resources.

Furthermore, we have also developed a URI anchor pad that, when mouse-clicked, directly issues a URI (Universal Resource Identifier) to the HTTP server to retrieve another composite pad from a remote site. This pad, when pasted on another pad, works as a link anchor in a hypermedia network spanned across the Internet. We have also developed the Mosaic-compatible pad that generates from any HTML (Hypertext Mark-up Language) string its corresponding Mosaic-compatible view as a composite pad. Each item in this Mosaic-compatible viewer is represented by an

independent pad. You can easily make a copy of any item or any page of this viewer as a pad to reuse it elsewhere.

6. Concluding remarks

The IntelligentPad architecture has several versions of its implementation. It was first implemented in Smalltalk 80, then in SmalltalkAgents, and also in C+ + using Interviews. The last one is open to the public for the evaluation of the IntelligentPad paradigm. Its source code can be downloaded from the ftp site ‘ftp.north.ad.jp’. Besides these systems, Fujitsu and Hitachi Software Engineering provide mutually compatible product versions for PC, Macintosh, and Unix workstations. They are cross-platform compatible systems. The development of the worldwide pad market used the SmalltalkAgents version of the IntelligentPad and the Macintosh Mosaic. This technology is already transferred to the product versions and the C+ +/Interviews version. The current implementation assumes that every site shares the same class library. This assumption requires the peri- odical update of the class library in each site. This update can be manually performed by accessing the on-line IntelligentPad system journal at the class-library-manager site through Mosaic. You can just click the system-update difference file name on a Mosaic page to get this file. The file transported by this mouse-click has another special file- name-extension ‘PCL’, which is mapped to an invocation of the class library update procedure.

The publication of pads into the worldwide repository is also an easy task for end-users. You can drag and drop your pad into a special pad called the PublicationPad. The PublicationPad saves this pad into a file in the local HTTP server. The file name of this file has ‘PAD’ as its extension.

Y. Tanaka i Information and Sofiware Technology 38 (1996) 201-21 I 211

Then it rewrites the local pad catalog written in HTML to include the name, the display image, and the description of this published pad. The updated catalog associates the name string and the image of this pad with the local file used to save the pad. They work as anchors when this catalog is viewed through the Mosaic system.

References

[ll

[21

[31

[41

[51

f61

[71

[81

R Dawkins The Selfish Gene Oxford University Press (1976)

M Stefik ‘The next knowledge medium’, The AI Magazine Vol7 No

1 (1986) pp 34-46

D C Engelbart ‘A conceptual framework for the augmentation of

man’s intellect’, Vistas in Information Handling Vol 1 Spartan

Books (1963) pp 1-29 D C Engelbart ‘Authorship provisions in AUGMENT’, Proc. of the

1984 COMPCON Con6 San Francisco (1984) pp 465-472

T H Nelson ‘A file structure for the complex, the changing, and the

intermediate’, Proc. of the ACM National Conference (1965) pp

84- 100

F G Halasz, T P Morgan and R H Trigg ‘NoteCards in a nutshell’,

Proc. of CHI+GI (1987)

B J Haan, P Kahn, V A Riley, J H Coombs and N K Meyrowitz

‘IRIS hypermedia services’, Comm. ACM Vol 35 No 1 (1992) pp

36-5 1

H Schiitt and N A Streitz ‘Hyperbase: a hypermedia engine based

on a relational database management system’, Proc. of the ECHT

‘90 (1990) pp 95-108

[91

[ 101

[ill

[ 121

[131

I141

[151

[161

[171

[181

[191

R M Akscyn, D L McCracken and E A Yoder ‘KMS: a distributed

hypermedia system for managing knowledge in organizations’,

Communications ACM Vol 31 NO 7 (1988) pp 820-835

L Hardman, D C A Bulterman and G Rossum ‘The Amsterdam

hypermedia model: extending hypertext to support real multi-

media’, Hypermedia Vol 15 No 1 (1993) pp 47-69

T Berners-Lee, R Cailliau, N Pellow and A Secret ‘The World-

Wide Web initiative’, Proc. INET ‘93 (1993)

M Andreessen ‘NCSA Mosaic technical summary’, NCSA Mosaic

Technical Summary 2. I (May 1993)

Y Tanaka and T Imataki ‘IntelligentPad: a hypermedia system

allowing functional composition of active media objects through direct manipulations’, IFIP ‘89 San Francisco (August 1989) pp

54 l-546

Y Tanaka ‘A toolkit system for the synthesis and the management

of active media objects’, Proc. of Deductive and Object-Oriented

Databases Kyoto (December 1989) pp 76-94

Y Tanaka ‘A synthetic dynamic-media system’, Proc. of Int. Conf

on Multimedia Information Systems Singapore (199 1) pp 299-3 10

Y Tanaka, A Nagasaki, M. Akaishi and T Noguchi, ‘Synthetic

media architecture for an object-oriented open platform’, in F H

Vogt (ed), Personal Computers and Intelligent Systems, Infor-

mation Processing 92 Vol III North-Holland (1992) pp 104-I 10

D C Smith, C Irby, R Kimball, W Verplank and E Harslem

‘Designing the Star user interface’, Byte, Vol 7 No 4 (1982) pp

242-282

R Rao and S Wallace ‘The X Toolkits’, Proc. of the Summer, I987

USENIX Conference (1987)

N S Borenstein Multimedia Applications Development with the

Andrew Toolkit Prentice-Hail (1990)