Hypermedia design, analysis, and evaluation issues

Design-oriented evaluation evolved out of our experi-ence both in developing several hypermedia applica-tions, and systematically inspecting and evaluatingmany commercial applications and prototypes. Weevaluate an application by its very nature, our methodaddresses neither software design (which can be eval-uated with general software evaluation techniques),nor how well the application relates to a domain or tospecific user needs (a main concern of other usabili-ty evaluation techniques). Our approach comple-ments more general evaluation methods [1, 11, 12,18, 19, 21] for the field of hypermedia.

In this article we describe our design-oriented eval-uation method and apply it to a highly popular com-mercial application: Microsoft’s Art Gallery, ahypermedia guide to the National Gallery in Lon-don’s painting collection. Art Gallery is an outstanding

and enjoyable application. Initially designed only forthe museum’s visitors [20], it is now widely availableas a CD-ROM [17].

We also discuss aspects of reuse in hypermediaapplications and propose some initial suggestions fordesigning for reuse.

Analysis DimensionsWe have identified several dimensions for analyzing ahypermedia application: content, structure, presenta-tion, dynamics, and interaction.

• Content: The pieces of information included in theapplication. These may consist of static (passive)media (such as formatted data, text strings,images, and graphics) or active (dynamic) media(such as video clips, sound tracks, and anima-

74 August 1995/Vol. 38, No. 8 COMMUNICATIONS OF THE ACM

F r a n c a G a r z o t t o , L u c a M a i n e t t i ,

a n d P a o l o P a o l i n i

Hypermedia Design,Analysis,and Evaluation Issues

ne can perform a heuristic evaluation of a hypermedia application effec-

tively by coupling a systematic analysis of the application based on a

hypermedia design model with general usability criteria, independent of

the specific application area, user profile(s), and user task(s). We call our

method design-oriented evaluation (as opposed to the user-oriented evalua-

tion commonly applied in usability testing), since it evaluates the internal

strength of the design underlying the hypermedia application.

tions). Content analysis may be the most impor-tant dimension in assessing a hypermedia applica-tion’s effectiveness. Content analysis, however,requires an application domain specialist and adeep knowledge of the users profiles and tasks; assuch, it falls outside design-oriented evaluation.

• Structure: The content’s organization. • Presentation: How application content and func-

tions are shown to readers. Presentation includesvisualization of individual granules of informationand of dynamic features, such as navigation. Forpassive media, presenting a granule of informationcorresponds to simple displaying; for active media,it corresponds to more elaborate playing.

• Dynamics: How users both interact with individualpieces of information (e.g., control the playing ofactive media) and move among them.

• Interaction: Using an application’s dynamic function-alities through operating on presentation elements.Most authors consider interaction a part of dynam-ics or presentation, as it blends both factors [12,15].

In this article we focus mainly on structure anddynamics, with some attention to interaction. (John-son also presents a design-oriented evaluation ofapplication structure in this issue.) Other authors inthis issue (Isakowitz et al., Kahn, Thüring et al.) exam-ine aspects concerning presentation and interaction.

Structural and Dynamic Elementsof a Design Model To perform a systematic application analysis, we needa design model to describe the application precisely.This use of design models is well known in the database area, where the popular Entity-Relationshipmodel [4], for example, is used both to develop newapplications and to document existing ones. We coulduse a number of hypermedia models, such as Lange’sEORM [16], Isakowitz et al.’s, or Schwabe and Rossi’s(both in this issue). Any model expressive enough foran effective and useful analysis will suffice. In this arti-cle we use the Hypermedia Design Model (HDM) todescribe the structure and dynamics of a hypermediaapplication, since it can describe complex multimediastructures and their dynamic properties. The sidebarpresents HDM’s main features and explains our termi-nology. We refer interested readers to the references[3, 5, 6, 7, 8, 9, 14] for a more complete presentation.

Heuristic Evaluation CriteriaWe could apply a huge number [12, 18, 19, 21] ofevaluation criteria, or heuristics [18] to our applica-tion, once we have analyzed and described it precise-ly. Here, we consider an adequate subset: richness,ease, consistency, self-evidence, predictability, read-ability, and reuse.

• Richness expresses the abundance of informationitems and ways to reach them.

• Ease measures information accessibility and howeasy to grasp operations are.

• Consistency measures application regularity, and canbe summed up a simple generic rule: Treat concep-tually similar elements in a similar fashion and con-ceptually different elements differently. Severalauthors consider consistency one of the mostimportant evaluation criteria [12, 15, 18, 19, 21].

• Self-evidence expresses how well users guess themeaning and the purpose of whatever (content ornavigational element) is being presented.

• Predictability expresses how well users anticipate anoperation’s outcome.

• Readability expresses the overall “feeling” about anapplication’s validity. Readability depends upon allfactors mentioned.

• Reuse considers using objects and operations in dif-ferent contexts and for different purposes. Reusepromotes consistency (and therefore predictability).Also, presenting the same material under differentperspectives and points of view enhances an appli-cation’s richness. Additional reasons for reuse canbe found among standard software engineeringargumentations: it reduces mistakes (debuggeditems generally can be reused safely), developmenteffort, and physical space required. The last is par-ticularly relevant for applications with largeamounts of multimedia information. Reuse greatlyexpands an application’s apparent size while mini-mally increasing its physical size.

An Analysis, Design and Evaluation ExampleHere we examine Microsoft’s Art Gallery CD-ROM.First, we show that a design model, such as HDM, candescribe an application designed with no knowledgeof the model itself. Second, we demonstrate that theresulting systematic description can help detect pos-sible problem areas within the application. Third, weshow the usefulness of applying the evaluation crite-ria described in the previous section.

We present a full description and analysis of ArtGallery in [6]. In this article, we only have room forsome of our analysis and evaluation examples. Wewish to stress, however, that the method and our lineof reasoning are more important than our specificconclusions; indeed, the reader is welcome to dis-agree with our interpretations.

Art Gallery: General Organization and ApplicativeLinks (Structure in-the-Large) Art Gallery is organized as a set of collections (in theHDM sense (see sidebar):

• Artists’ Lives: a collection of entities of type“Painter,” arranged in alphabetical order by thepainter’s name. An index (see Figure 1) allowsone to select a painter using his or her name.

• Historical Atlas: a collection of “City Collec-tions.” Each city collection (e.g., Florence) is in

COMMUNICATIONS OF THE ACM August 1995/Vol. 38, No. 8 75

turn a collection ofentities of type “City-Period” (e.g., “Flo-rence 1300”).

• Picture Types: a col-lection of entities oftype “Picture Type”(e.g., “Altarpieces”).

• General References: acollection of entitiesof type “Subject”(e.g., “Tempera”), arranged in alphabetical order.

• Guided Tours: a collection of four lectures. Eachlecture is a collection of nodes selected fromother entities, presented from a specific point ofview (e.g., “Composition and Perspectives”).

An additional collection---History---is generatedas the user explores the application. History con-

tains all nodes visited at any given time. Entity types include the following:

• “Painter” has a root component providing infor-mation about the painter and a number of paint-ings (see Figure C in the HDM sidebar). The rootcomponent consists of one to three nodes of type

• “Artists’ Lives”; the paintings are accessible fromthe first node of the root. Each painting is a com-ponent consisting of a variable number of nodesof type “Painting” (see figures A and B). Figures

2–4 show the threeroot nodes of painterPiero della Francesca.Figure 5 shows anexample node of type“Painting.”

• “Picture-Type” alsohas a root compo-nent and a numberof paintings. Its rootcomponent compris-es nodes of type “Pic-ture Type.”

• “Subject” also has aroot componentand a number ofpaintings. Its rootcomprises one node(or occasionally

more) of type “General Reference.” Figure 6 pro-vides an example, “Tempera” (which is a paint-ing technique).

Applicative links (see sidebar) express relationshipswith application-specific semantics. We divide applica-tive links into two broad categories: schema links andgeneric links. Schema links have predefined syntactic


Fig. 1

Fig. 2 Fig. 3

and semantic features. Generic links vary dependingon the context in which they are used. As such, gener-ic links are less standardized than schema links.

Consider some of Art Gallery’s schema links. Everynode of type “Painting” is connected to several items(see Figure 7) constituting relevant, related topics: itspainter, city-period, and picture-type, as well as (pos-sibly several) subjects. There are no links from apainting to the guided tours that include it, althoughat the end of each guided tour there are links to eachpainting mentioned in it.

Every node of type “Painter” is connected to sever-al items (see Figure 8): all city-periods in which theartist or one of his paintings is mentioned, all picture-types including one of his paintings, and all subjectsto which one of his paintings is related. No links go toor come from guided tours. Similar schema links aredefined for the other entity types.

Now we examine some aspects of the applicationin light of our evaluation criteria, starting withschema links. The painting “The Baptism of Christ”by Piero della Francesca is included in subject “Tem-pera” (within collection “General References”), asshown in Figure 6. Therefore, as one of his paintingsis included in that subject, painter Piero dellaFrancesca is correctly connected to “Tempera,” asshown in Figure 8. However, Figure 7 shows thatthere is no schema link from painting “The Baptismof Christ” to “Tempera.” Additionally, despite the factthat subjects “Jesus Christ” and “Baptism of Christ”also include painting “The Baptism of Christ,” its

painter Piero della Francesca has no schema links tothese subjects. A casual user may not notice thisinconsistency, but a user systematically exploring theapplication not only may be puzzled by it, but alsomay miss information (e.g., that “The Baptism ofChrist” employs the tempera technique).

Next we discuss one facet of generic links. In Figure3, Piero della Francesca is linked to DomenicoVeneziano and to one of his paintings (since the twoartists worked together), to Paolo Uccello and one ofhis paintings (because Uccello was a source of inspi-ration to Piero), and to Signorelli (who was one ofPiero’s pupils). Now, generic links are not intended tofollow predefined patterns, so it makes no sense tolook for consistency (or inconsistency) among them.The problem is that every time the reader encountersone, he or she must guess its meaning. One couldargue that the relationship “pupil-of” could have beentreated as a schema link, but this would lead to a dis-


Hypermedia Design, Analysis, and Evaluation Issues

Figure 1. Index of paintersFigure 2. First of three nodes of the root compo-nent of painter Piero della Francesca; miniaturizedpainting images represent structural links to Piero’spaintingsFigure 3. Second of three nodes of the root com-ponent of painter Piero della Francesca; a number ofgeneric links are embedded in the textFigure 4. Last node of the root component ofpainter Piero della FrancescaFigure 5. Third of six nodes of “The Baptism ofChrist”, first of four painting components of painterPiero della Francesca

Fig. 4 Fig. 5

Slot (in-the-small)STRUCTURE: An atom-ic piece of information.Slots can have a simpletype (e.g., an integer) orcomplex type (e.g., avideo synchronized withsound).DYNAMICS: Interactingwith the slot’s content.Multimedia slots, forexample, have complexfeatures such as controlvideo play [10].

Frame (in-the-small)STRUCTURE: An aggre-gate of slots meant toform a cohesive granuleof information. A frame isa presentation unit (i.e.,what is shown to the user).DYNAMICS: Presentinga frame’s slots. Passivemedia slots are normallypresented at once.Active media slots mayrequire complex strate-gies synchronizing paral-lel or sequencedpresentations [2, 13].

Node (in-the-small)STRUCTURE: A naviga-tional unit. In this articleeach node is associatedwith one unique frame.

Node type (in-the-small,in-the-large)Nodes of the same typehave similar frames andconnections (see Link).

Component (in-the-small)STRUCTURE: A groupof nodes constituting a

logical unit. In ArtGallery, for example, acomponent called“Painting” consists of sev-eral nodes, as does “Rootof a Painter” (Figure A).DYNAMICS: Navigatingacross the component’snodes (see Figure A).

Component type (in-the-small, in-the-large)Components of thesame type have similarnodes, dynamics, andconnections.

Entity (in-the-large)STRUCTURE: A groupof components corre-sponding to a real-worldobject. Figure B showsPainter entity “Pierodella Francesca,” com-prising a root compo-nent and severalPainting components.DYNAMICS: The naviga-tion determined by theentity’s structure (seeStructural links).

Entity-type (in-the-large)Entities of the same typehave similar structures(Figure C) and connec-tions.

Collection (in-the-large)

STRUCTURE: A set ofobjects. Collection mem-bers [5] can include enti-ties, components, singlenodes and other (nested)collections. A collectionrepresents a taxonomy(e.g., all paintings on a

given subject) or simplygroups objects toimprove application read-ability. Often a collection-node provides a previewof the collection withaccess paths to its mem-bers (see Index links).DYNAMICS: The naviga-tion determined by thecollection’s links.

Link (in-the-large)STRUCTURE: A connec-tion between two differ-ent objects (i.e., entities,components, nodes orcollections).DYNAMICS: Traversingthe link, i.e., movingfrom the source to thedestination, and present-ing and activating thedestination. The destina-tion is always a node, pos-sibly representing alarger granule (e.g., anentity). Defining an appli-cation’s navigation stylecorresponds to definingthe rules for traversinglinks [8]. Link traversalrequires extensions foractive, time-dependentmedia [13]. For example,authors must define thestate in which to leave thesource (e.g., reset to theoriginal state or leave asmodified by activation).

Structural links (in-the-large)STRUCTURE: Structurallinks tie together a com-ponent’s nodes or anentity’s components,

reflecting the logicalinterrelationships amongthe nodes or compo-nents (see Figure A).DYNAMICS: Structuralnavigation, i.e., travers-ing structural links.

Collection links (in-the-large):STRUCTURE: Collec-tion links (either indexlinks or guided tourlinks) tie together thecollection-node and col-lection members accord-ing to the collection’slogical structure.DYNAMICS: Collectionnavigation, i.e., travers-ing collection links.

Index links (in-the-large)STRUCTURE: Indexlinks connect the collec-tion node to each mem-ber of the collection,and vice versa.DYNAMICS: Traversingfrom an index (collec-tion node) to a member,or vice versa.

Guided Tour links (in-the-large)STRUCTURE: Guidedtour links connect a col-lection’s members [8,22] in a linear sequence(for linear collections),with each member con-nected to the next andprevious one. In circularcollections, the last mem-ber connects to the first.DYNAMICS: Guidedtour navigation moving


The HDM Model

HDM [3, 5, 6, 7, 8, 9, 14] is a model for designing hypermedia applications. Over 20 development groups in six countries have

used it (or some dialects of it) for applications in different domains. HDM is under constant development, evolving along

with the hypermedia field. Here we informally define the subset of HDM concepts used to describe Microsoft’s Art Gallery.

In-the-large: The overall application organization and behaviorIn-the-small: Details about the application’s organization and behaviorStructure (in-the large, in-the-small): The organization of the application’s contentDynamics (in-the large, in-the-small): Application behavior

among members direct-ly, without returning tothe collection node.Readers can executeguided tours manually orautomatically (i.e., somecontent- or time-depen-dent synchronizationinvokes traversal to thenext member). Activemedia slots often invokecontent-dependent syn-chronization (e.g., theend of an audio trackinvokes traversal to thenext member).

Applicative, schema andgeneric links (in-the-large)STRUCTURE: Applica-tive (either schema orgeneric) links connecttwo objects according tosome application relation-ship. Schema links belongto a defined link type;generic links do not.DYNAMICS: Applicativenavigation, i.e., traversingschema or generic links.

Link type (in-the-small,in-the-large)Link types group allschema links with a simi-lar meaning, therebyconnecting objects ofthe same category (e.g.,a painter with a histori-cal period or a paintingwith a subject). C


RootComponent

Componentsof type“painting”

PF.2 PF.3

p1.1 p1.4 p 2.1 p3.1 p4.1

PF.1, 2, 3 : the nodes of the root component of Piero della Francescap i.j : j-th node of the i-th painting component

“Next Page” Link(Embedded) Structural Link among nodes of different components

PF.1

Piero dellaFrancesca

RootComponent

Componentsof type“painting”

The Baptismof Christ

St. Michael The Nativity A Battle

ROOT

PAINTING PAINTING

Figure B. Structure in-the-large of entity “Pierodella Francesca” of type “Painter”

Figure A. The detailsof entity “Piero dellaFrancesca”

Figure C. Structure in-the-large of entity type “Painter”

the bottom of the screen. From the last member of acollection, Next Page usually leads to the first (defin-ing a “circular collection”).

We can observe a number of missing navigationpatterns often found in similar applications, e.g.,backward scan of a collection, direct jump to a collec-tion’s last or first member, and jump from a memberto the index (to review the list of members). Leavingthese out does simplify the interface, and simple inter-action is desirable. But it results in a lack of richnessof navigation features. Limited collection navigation

can frustrate the userperforming a well-defined search taskrequiring extensive navi-gation. Readers justbrowsing around forinteresting informationwith no specific task inmind probably will noteven notice the lack of


cussion about the specific content of the application,which gets beyond the scope of this article. From anengineering point of view we could apply a statisticalmeasure: if most painters have a pupil link, then makeit a schema link otherwise leave it as a generic link.

Collection Navigation (Dynamics in-the-Large)A collection is an ordered set of objects called itsmembers (see sidebar). In general, one can navigate acollection in two ways (separately or together): direct-ly accessing a member (index navigation) or movingfrom one member tothe next according tothe collection’s ownordering (guided tournavigation).

In Art Gallery, onemoves from one mem-ber to the next with theNext Page button, locat-ed in the control bar at

Figure 6 . The root component of entity “Tem-pera” (of type Subject); miniaturized paintings rep-resent the structural links to paintingsFigure 7. View of schema links outgoing from “TheBaptism of Christ”; first of four painting compo-nents of painter Piero della FrancescaFigure 8. View of schema links outgoing from theroot component of painter Piero della Francesca

Fig. 6

Fig. 7 Fig. 8

richer navigation fea-tures. Which usage ismore relevant for ArtGallery’s users (orga-nized access or leisurelybrowsing) goes beyondour scope.

We also observe anumber of inconsisten-cies. Within each guid-ed tour, navigationalways starts from the first member of the collection.Readers usually select Next Page to scan forward inthe collection. From the last member of a guidedtour, Next Page leads to an index from which anypainting mentioned in the guided tour can bereached. For consistency with the other collections(always “circular”), we expected to be able to go to

the first member of theguided tour. Also, guid-ed tours are a kind oflinear tale too long toassimilate at once; oftenone wants to start againfrom a specific point ofthe tale. The lack ofdirect access to anymember of a guidedtour is particularly

annoying; one must always return to the start andrepeatedly apply Next Page.

Multimedia interaction presents another inconsis-tency. Guided tour nodes contain an audio comment,which is played automatically (with possibility of turn-ing it off) each time the node is entered. Entering aguided tour from an index automatically activates the


Figure 9. “The Supper at Emmaus” (by Caravaggio),sixth of eight nodes of guided tour “Compositionand Perspective”; generic links are available, butschema links (See Also) are notFigure 10. First member of guided tour “Composi-tion and Perspective”; compare with Figure. 5Figure 11. “The Martyrdom of St. Sebastian (byPollaiolo)”, second of eight members of guided tour“Composition and Perspective”

Fig. 9 Fig. 10

Fig. 11


audio comment. When the audio finishes, one auto-matically jumps to the first member, and its audioactivates. But transition to the next member within aguided tour, inconsistently and unexpectedly, is man-ual (Next Page), not automatic.

Now consider the lack of richness in navigating theHistory collection, the set of nodes previously visited.Users only can access the History backwards, throughthe Go Back button. Furthermore, access is destruc-tive, discarding the last node (and progressivelydestroying the session’s history). Users may prefer, forexample, to review a list of previously visited nodes andselect any one, without removing it from the History.

More serious, we find that readers often interpretGo Back as a “go-to-previous” collection link, i.e., theopposite of Next Page. True, going back over a num-ber of Next Pages in the same collection gives thisimpression, but in general, Go Back is not the reverseof Next Page. Thus, Go Back often is not self-evident.

Art Gallery has other unpredictable behaviors.Assume the reader directly accesses “Master of theAachen Altarpieces” from “M” in the alphabeticalindex of painters and then uses Next Page to reachthe following painter in alphabetical order. Surpris-ingly, he or she gets the painter Hans von Aachenfollowed by other painters with initial letter “A.”This results from using alternative names forpainters in the index. Each painter, however,appears only once in the collection Artists’, Lives,corresponding to a particular alternative name. Inthis example, the alternative name “AACHEN Altar-piece Master” determines the position within theArtists’ Lives collection.1

Structural and Applicative Link Navigation(Dynamics in-the-Large)Structural navigation (see sidebar) concerns explor-ing nodes belonging to the same entity. Considerstructural navigation for the entity type “Painter.”

Readers can traverse a painter’s root nodes (fig-ures 2, 3, 4, and sidebar Figure A ) forward with NextPage, but no Previous link is provided. From the lastnode of a painter’s root, Next Page leads to the fol-lowing painter. Only the message in the control bartells the reader when Next Page takes him to a differ-ent painter, resulting in substantial confusionbetween collection navigation (i.e., moving amongpainters) and structural navigation among one enti-ty’s nodes. While a single button (Next Page) has theadvantage of simplifying the interface, it does over-load the button with several different meanings.

From the first node of a painter (see Figure 2), thereader visualizes the collection of the entity’s paint-ings and can access any member via structural links(see Figure A). Structural links are represented by

embedding miniaturized pictures of the paintings inthe node. We find it confusing to represent structur-al links, which keep the entity together, in a mannersimilar to generic links (see Figure 3), which usuallyare semantically weaker and less important (i.e.,inconsistent treatment).

Readers can explore each of a painting’s nodes(forward using Next Page). A message in the controlbar describes the reader’s current position within theentity (see Figure 5, showing the third node of thefirst painting of Piero della Francesca). When all of apainting’s nodes have been traversed (e.g., the read-er reaches the sixth node of the first painting by Pierodella Francesca), Next Page leads to the next paint-ing of the same painter. From the last node of the lastpainting, Next Page leads to the first node of the firstpainting. Using Next Page only, it is impossible to exitthe entity, while at root level Next Page moves thereader from one painter to the next.

The following link inconsistency puzzles us: Whilea structural link (presented as an embedded linkplace-holder) leads from a painter to its paintings,the opposite connection, from a painting to itspainter (e.g., from Figure 5 to the Piero dellaFrancesca root, represented in figures 2--4), is anapplicative link (See Also button).

Although, by definition, more important thangeneric links, in Art Gallery schema links are lessaccessible. Generic links are embedded in the text(see Figure 3) in such a way that they attract the read-er’s attention; schema links are accessible throughthe misleading See Also button, suggesting a lessimportant connection (see figures 7 and 8). We clas-sify this as a lack of self-evidence, or lack of consis-tency between the interaction’s semantics and itspresentation style.

When inserted in a guided tour, a node’s schemalinks are excluded, while the relatively unimportantgeneric connections are still available. For example,in Figure 9’s node “The Supper at Emmaus” withinthe guided tour “Composition and Perspectives,” theSee Also button is unavailable and thus all the infor-mation related to the artist Caravaggio and this paint-ing are not directly accessible. It is possible, however,to jump to Annibale Carracci via a generic link, as thisartist was a possible source of inspiration for Caravag-gio! We can see that the choice of not properly rep-resenting structural links (i.e., from paintings topainter) which has been noticed before, shows itsnegative consequences.

Art Gallery in-the-SmallAs a case study of Art Gallery’s detailed features (“in-the-small”), we focus on nodes concerning paintings.Information about a painting is organized in severalnodes, with some fixed slots (fields), i.e., a title (thename of the painting) and a label (“Paintings”).Optional slots are of type text, image, image with cap-tion and animation.


1 A minor inconsistency is that the alphabetical index of painters containsthe sequence Piero di Cosimo, Piero della Francesca and Pinturicchio (seeFigure 1), while the actual sequence in the collection Artists’ Lives is Pierodella Francesca, Piero di Cosimo and Pinturicchio.



Figure 12. “The Battle of San Romano”(by Paolo Uccello), fourth of eight mem-bers of guided tour “Composition and Per-spective”; compare with Figure 13.

Figure 13. “The Bat-tle of San Romano”;first of seven nodes ofthe second paintingcomponent of painterPaolo Uccello; comparewith Figure 12.

Consider “The Baptism of Christ,” by Piero dellaFrancesca. Figure 5 shows the third of the six nodesdedicated to this painting. The structure in-the-smallof this node consists of a title (“The Baptism ofChrist”), the label “Paintings,” a column of text(describing its geometric features), a still image show-ing the painting, and a button labeled “ANIMA-TION.” All four passive media slots are presented(displayed) simultaneously, but the animation is pre-sented (played) upon user request. The only possibledynamic in-the-small is activating and deactivatingthe animation. Activation draws a number of linesand circles over the painting (in approximately 25seconds), effectively showing the painting’s geomet-ric features. At the end of the animation all the linesdisappear. During execution, clicking anywhere stopsthe animation and removes all lines.

Now consider a separate node:the first member of the guidedtour collection “Compositionand Perspectives,” shown inFigure 10, which correspondsto the third “page” of “TheBaptism of Christ” shown inFigure 5. It has an additional

(sixth) slot---an audio comment. Also, the label slotnow reads “GUIDED TOURS.” From the contentpoint of view, the audio slot has the important roleof providing the motivation for including the paint-ing in the guided tour. This an excellent usage foraudio: integrating a node in a context differentfrom its original purpose. This also exemplifiesreusing a node in a different context (from the com-ponent painting of a painter to a guided tour).

Consider the audio in terms of dynamics in-the-small. Audio starts automatically upon entering thenode. The reader can stop it by clicking anywhereand restart it by clicking on the label “GUIDEDTOURS,” which contains a loudspeaker icon. Theanimation activates as before and can be started whilethe audio plays.

Major dynamics in-the-small inconsistency prob-lems arise, since the interaction between audio andanimation creates unpredictable effects. Sometimes,when the animation is active, the activation of theaudio is precluded. Other times, once the activationis started, clicking on the audio loudspeaker iconstarts the audio comment. Activating the audio, how-ever, does not preclude activating the animation. Theanimation can be stopped by clicking anywhere,including the control bar and audio label. The audiocan be stopped by clicking anywhere except the ani-mation button (which starts the animation) or on thecontrol bar (which executes the command corre-sponding to the clicked button). If audio and anima-tion are both activated, the first click on the page(even on the control bar!) stops the animation, but asecond click is needed to stop the audio. The lack of

self-evidence, predictability and readability in theinteraction is quite obvious; the reader must (as wedid) learn how to use the active media in this node bytrial and error.

The second step of the guided tour “Compositionand Perspectives” presents Pollaiolo’s painting “TheMartyrdom of St. Sebastian,” shown in Figure 11.Focusing on the dynamics in-the-small of this nodereveals some inconsistencies with respect to the pre-vious picture, “The Baptism of Christ.” It has two but-tons, labelled “ANALYSIS” and “PAINTING.”Activating the ANALYSIS button draws lines over thepicture (in a few seconds). In contrast to “The Bap-tism of Christ,” the lines remain when the animationfinishes. Clicking anywhere while lines are beingdrawn immediately produces the final set of lines,instead of removing them as before. Once activated,clicking the ANALYSIS button has no effect whatso-ever, instead of restarting the animation. Clicking thePAINTING button erases any lines drawn (in a fewseconds). The audio works almost as for “The Bap-tism of Christ,” except that activating the animationnever precludes playing the audio.

Outside of the guided tour, in the third node ofthe same painting “The Martydom of St,4. Sebastian”,the animation button starts a very interesting shortsequence of small pictures, which remain on thescreen when the sequence is finished. Clicking any-where suspends the animation at the positionreached; clicking on the animation button restarts it.However, apart from returning to this node from theprior page, there is no way to recover the originalimage without pictures overlaid.

Clearly, this lack of consistency in interacting withapparently similar features in different nodes createsproblems with predictability, and therefore readabili-ty. Readers must try several options before gettingwhat they want, if they get it at all.

Comparing the two corresponding nodes in fig-ures 5 and 10 reveals another strange presentationinconsistency. First, different line lengths make thetext slots look different, despite having exactly thesame content. Readers may never notice the inconsis-tency,2 but we are curious about the editorial processthat produced such an anomaly.

We conclude this section with some issues con-cerning reuse. Consider the fourth node of the guid-ed tour “Composition and Perspective,” illustrating“The Battle of San Romano” by Paolo Uccello (seeFigure 12). Comparing this page with the “original”(see Figure 13) reveals different text in each. In thenode of entity type “Painter,” the last paragraphexplains what will be seen in the following six nodesconcerning “The Battle of San Romano.” Since theguided tour contains only two nodes of that painting,that paragraph is no longer appropriate and, correct-


2 Actually, we did not notice it either; an anonymous reviewer brought thisto our attention.

ly, has been removed. This example clearly illustratesproblems reuse can create: Sometimes values associ-ated with some slots must be modified for the newcontext. We suggest that the author concentrate allcontext-dependent information in a few slots (or bet-ter yet, just one slot). Then reuse requires deleting orreplacing just those context-dependent slots.

Lack of self-sufficiency can be another obstacleto reuse. In the second node of the guided tour“Composition and Perspective” (see Figure 11), theauthor’s name does not appear, most probablybecause in the original context this would be obvi-ous. Only the sound slot provides, among otherinformation, the painter’s missing name. Buildingupon this, we extend our previous suggestion.Reusing a node could benefit from providing addi-tional slots (audio, in the case of the Art Gallery) tomake nodes more readable in a context other thanthe one for which it was conceived.

ConclusionsPrecise design primitives not only improve the hyper-media design and development process (as discussedin [3, 5, 7, 9] as well as Isakowitz et al., and Schwabeand Rossi in this issue), but also facilitate analyzingand evaluating hypermedia applications. Designprimitives enable a systematic and precise descriptionof an application. Proceeding from description toevaluation requires evaluation criteria. The resultingevaluation is design-oriented, concerning primarilythe internal strength of the design and neglecting therelationship with user profiles and tasks.

Of course, a user-oriented evaluation (as typicallycarried out in usability tests [1, 18, 19, 21]) can par-tially modify or even reverse a design-oriented evalu-ation’s recommendation. A technical inconsistency,for example, could be judged acceptable, or evendesirable for some users and application domains.Yet we believe that more often than not the two eval-uation methods will agree. In any case, a preliminarydesign-oriented evaluation could detect possibleareas of concern and better focus the usability tests.

We introduced a number of evaluation criteriaderived from our experience and from the literature:richness, ease, consistency, self-evidence, predictabil-ity, readability, and reuse. Other analysts may choosedifferent criteria and still retain the flavor of ourapproach. The topic of hypermedia evaluation crite-ria itself would prove an interesting subject forresearch and debate. We know authors of excellenthypermedia applications who would rather sacrificeconsistency and richness of navigation features, infavor of a more elegant layout design (though weprobably would not). In any case, sacrificing one eval-uation criterion in favor of another should be a con-scious, well-motivated, and justifiable decision.

Our final point concerns reuse. While reuse intraditional software engineering mainly affects pro-ductivity, in hypermedia applications it affects an

application’s completeness and readability. Inhypermedia, a granule of information can beencountered on different access paths, in differentcontexts, and from different points of view. Thisgreatly enhances the flow of the information for thereader, and thereby application richness. Reuse,however, creates problems for authors, especiallywith multimedia data, which, as we saw, can causecommunication channel conflicts. If, for example,Art Gallery paintings had their own audio comments,it would have been difficult to add an audio com-ment for a guided tour and coordinate interactionbetween the two audio tracks. Perhaps for this rea-son, most applications we have encountered thatexploit reuse have very simple multimedia features.Applications with rich multimedia tend to have littlereuse. The field should develop models and, aboveall, methodologies for increasing reuse, and therebymaking multimedia applications more efficient andeasier to develop.

References 1. Bearne, M., Jones, S. and Bearne J. S-F. M. Towards usability

guidelines for multimedia systems. In Proceedings of Multimedia‘94 ( S. Francisco, Oct. 1994), ACM, New York, 1994, 105--110.

2. Buchanan, M.C. and Zellweger, P. Automatic temporal lay-outmechanisms. In Proceedings of Multimedia ‘93 (Anaheim, Calif.,Aug. 1993), ACM, New York, 1993, 341--350.

3. Cavallaro, U., Garzotto, F., Paolini, P., and Totaro, D. HIFI:Hypertext interface for information systems. IEEE Software 10, 6(Nov. 1993), 48--51.

4. Chen, P. The entity-relationship approach: Toward a unifiedview of data . ACM Trans. Data Base Syst. 1, 1 (Jan. 1976), 9--36.

5. Garzotto, F., Mainetti, L. and Paolini, P. Adding multimediacollections to the Dexter model. In Proceedings of ECHT’94 -European Conference on Hypermedia Technologies (Edinburgh, UK,Sept. 1994), ACM, New York, 70--80.

6. Garzotto, F., Mainetti, L. and Paolini, P. HDM analysis of theMicrosoft Art Gallery CD-ROM. Tech. Rep. DEI#35-94 (Politec-nico di Milano, Sept. 1994).

7. Garzotto, F., Mainetti, L. and Paolini, P. Hypermedia Applica-tion Design: A Structured Approach. In Designing User Interfacesfor Hypermedia, W. Schuler, J., Hannemann, and N. Streitz, Eds.Springer Verlag, 1995 (in press).

8. Garzotto, F., Mainetti, L., and Paolini, P. Navigation in hyper-media applications: Modeling and semantics. J. Organ. Comput.(in press).

9. Garzotto F., Paolini P., and Schwabe, D. HDM - A Model BasedApproach to Hypermedia Application Design. ACM Trans. Inf.Syst., 11, 1 (Jan. 1993), 1--26.

10. Gibbs, S., Breiteneder, C. and Tsichritzis, D. Data modeling oftime-based media. In Proceedings SIGMOD’94 (Minneapolis,May 1994,), ACM, New York., 91--102.

11. Hardman, L. Evaluating the usability of the Glasgow onlinehypertext. Hypermedia 1, 1, (Spring, 1989) 34- 63.

12. Hardman, L., and Sharrat, B. User-centred hypertext design:The application of HCI design principles and guidelines. InHypertext State of the Art, R. McAleese and C. Green., Eds. Intel-lect 1990, 252--259.

13. Hardman, L., Bulterman, D.C.A. and Van Rossum,G. The Ams-terdam hypermedia model: Adding time and content to theDexter model. Commun. ACM, 37, 2 (Feb. 1994), 50--62 .

14. HIFI Team. HDM+ User Manual. Deliverable 6, ESPRIT ProjectHIFI, 1993.

15. Hodges, M. and Russell, M. Multimedia computing--Case studiesfrom MIT Project Athena. Addison-Wesley, Reading, Mass.,1993.

16. Lange, D. An object-oriented design approach for developing

C



hypermedia information systems. J. Organ. Comput. (in press). 17. Microsoft. Art Gallery CD-ROM—The Collection of the National

Gallery, London. Microsoft Corporation, 1993.18. Nielsen, J. Usability Engineering. Academic Press, 1993. 19. Preece, J. Human-Computer Interaction. Addison-Wesley, Read-

ing, Mass.,1994. 20. Rubinstein, B. The Micro Gallery at the National Gallery. Tech.

Rep., Cognitive Applications Limited, Brighton, UK, 1993. 21. Shneiderman, B. Designing the User Interface. Addison-Wesley,

Reading, Mass., 1987. 22. Trigg, R.H. Guided tours and tabletops: Tools for communi-

cating in a hypertext environment. ACM Trans. Off. Inf. Syst.6,4 (Oct. 1988), 398--414.

About the Authors: FRANCA GARZOTTO is a research associate at the Department ofElectronics and Information, Politecnico di Milano, where shepresently serves as Managing Director of the Hypermedia Labora-tory. Her current research interests include hypermedia applica-tion modeling, hypermedia authoring systems, and multimediaevaluation. She has been involved in various European researchprojects in these fields, within the ESPRIT program.

LUCA MAINETTI is a Ph.D. student in computer science andresearch assistant at the Department of Electronics and Informa-tion, Politecnico di Milano. Current research interests includeobject oriented systems, multimedia technology, multimedia databases, and multimedia development tools.

PAOLO PAOLINI is currently a full professor at University ofLecce (Italy) and Scientific Director of the Hypermedia Laborato-ry at the Department of Electronics and Information, Politecnicodi Milano. Current research interests include hypermedia engines,hypermedia modeling, hypermedia authoring systems, and multi-media application development tools. He has been scientificallyresponsible for various European research projects in these fields,five within the ESPRIT program. He is currently Associate Editor ofACM’s Transactions on Information Systems.

Authors’ Present Address: Department of Electronics andInformation, Politecnico di Milano, Via Ponzio 34/5, 20133,Milano, Italy; email: {garzotto, mainetti, paolini}@elet.polimi.it

Copyright of the CD-ROM engine of Microsoft Art Gallery —The Collectionof the National Gallery, London by Cognitive Applications Limited(Brighton, UK.) For more information, see copyright information within CDbooklet.

Permission to make digital/hard copy of all or part of this material withoutfee is granted provided that copies are not made or distributed for profit orcommercial advantage, the ACM copyright/server notice, the title of thepublication and its date appear, and notice is given that copying is by per-mission of the Association for Computing Machinery, Inc. (ACM). To copyotherwise, to republish, to post on servers or to redistribute to lists, requiresprior specific permission and/or a fee.

©ACM 0002-0782/95/0-800 $3.50


Documents

Hypermedia design, analysis, and evaluation issues