Checking Consistency in Multimedia Synchronization Constraints

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 6, NO. 4, AUGUST 2004 565

Checking Consistency in MultimediaSynchronization ConstraintsHuadong Ma, Member, IEEE, and Kang G. Shin, Fellow, IEEE

Abstract—Constraint-based design is often used to correctlyauthor a multimedia scenario due to its flexibility and efficiency.However, such a system must provide a mechanism with whichusers can easily manipulate the underlying structures to meet theapplication requirements. This paper proposes a novel methodfor analyzing multimedia synchronization constraints based onthe constraint graph and classification, which is essential indeveloping efficient system support tools for constraint-basedauthoring systems. We specify temporal and spatial relationsbetween multimedia objects, and use a directed graph to repre-sent the constraints among the objects in a multimedia scenario.Moreover, we develop a method for analyzing temporal andspatial synchronization constraints based on graph theory, solvingthe problems of completeness checking, consistency checking,constraints relaxation and automatic spatio–temporal layoutgeneration in a unified theoretical framework. We also discussthe effects of user interactive authoring. Compared with othermethods, the proposed approach is simpler, more efficient, andeasier to implement.

Index Terms—Consistency checking, graph, multimedia au-thoring, synchronization constraint.

I. INTRODUCTION

MULTIMEDIA systems must be able to capture, store,and transfer multimedia data and compose sophisticated

multimedia presentations. One of the peculiar characteristics ofmultimedia data is that they are tightly coupled through bothspatial and temporal constraints. Thus, the key problem in multi-media systems is the modeling of spatial and temporal synchro-nization. The temporal relation is considered as the master rela-tion in a multimedia scenario, and the spatial relation is a slaveto the temporal relation. Thus, we first focus on temporal syn-chronization, and then discuss both synchronizations together.

From the viewpoints of temporal synchronization, a mul-timedia system essentially consists of two components: thetemporal authoring system and the temporal control system(scheduler). The temporal authoring system lets a user definewhat each multimedia presentation looks like. The scheduler

Manuscript received September 29, 2000; revised August 13, 2002. This workwas supported in part by the U.S. National Science Foundation under Grant EIA-9806280, the Chinese NSFC under Grants 69873006 and 60242002, UNU/IISTOffshore Project, and the Chinese EYTP of MOE. This work was performed inpart while H. Ma visited The University of Michigan and parts of this work werepresented at the IEEE ICME2000. The associate editor coordinating the reviewof this manuscript and approving it for publication was Prof. Wayne Wolf.

H. Ma is with School of Computer Science & Technology, Beijing Uni-versity of Posts and Telecommunications, Beijing 100876, China (e-mail:[email protected]).

K. G. Shin is with the Real-Time Computing Laboratory, Department of Elec-trical Engineering and Computer Science, The University of Michigan, AnnArbor, MI 48109-2122 USA (e-mail: [email protected]).

Digital Object Identifier 10.1109/TMM.2004.830807

deals with the control of multimedia presentation, includingintramedium and intermedia synchronizations.

The temporal relations in multimedia systems define thetemporal dependencies between media objects. Temporal re-lations can be specified by two basic models: timeline modeland constraint-based model. The constraint-based model has aclear advantage over the traditional timeline approach due to itsflexibility and efficiency. Despite their various advantages, theconstraint-based authoring systems must provide a mechanismfor users to easily manipulate the underlying structures so asto implement completeness checking, consistency checking,constraints relaxation, and layout generation.

The problem of temporal synchronization was addressed in[3] and [4]. According to the representation methods of tem-poral constraints, a model can be classified as a graph model[5], [19] a Petri-net based model [12], [18], an object-orientedmodel [10], or a language-based model [2], [7], [13], [15]. In agraph model, constraints are represented as edges in the graph,whereas the Petri-net model captures the constraints by meansof places and transitions. In the object-oriented model, the con-straints are modeled as object attributes, whereas the language-based model uses programming language constructs to expressconstraints.

An important issue, which has not yet been fully investigated,is consistency checking. Especially for multiuser collaborativeauthoring of a multimedia scenario, there usually exist incon-sistencies in the constraints specified by different users. Even ifchecking the consistency of the specification is very importantfor producing a correct presentation, only a few methods, suchas Firefly [5], TCSP [2], and ETM [19], explicitly provide mech-anisms for specification verification. Specification checking inTCSP is performed by means of the TCSP proof theory, whereasFirefly provides a schedule from the specification. If no pre-sentation schedule can be generated, then an error message isissued, listing all the events which caused the inconsistencies.The development of dynamic consistency checking mechanismsis also an important topic: in this approach, the consistencychecking is performed online, thus providing immediate feed-back to the user when s/he adds an invalid constraint. The ETMuses a temporal constraint network in which nodes represent thestart and end times of an object, and edges represent the dif-ference constraints to define the temporal relationships amongmedia objects, and provide an efficient incremental constraintsolver to check for anomalies and generate a temporal schedule.In addition, [16] also assumes a temporal constraint network forchecking the temporal integrity of interactive multimedia docu-ment scenarios.

1520-9210/04$20.00 © 2004 IEEE

566 IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 6, NO. 4, AUGUST 2004

Another important issue is how to adapt when the satisfac-tion of synchronization constraints cannot be guaranteed, orwhen an inconsistency is found. In this case it may be usefulto systematically relax the constraints in order to derive afeasible presentation. The simplest way is to explicitly re-quire the insertion or deletion of certain constraints from thespecification. Alternatively, constraints can be divided intomandatory constraints and optional constraints that would bebetter to meet but can be ignored with acceptable performancedegradation. Another approach is to define and follow theorder of relaxing the constraints. Firefly [5] provides someform of relaxation by shrinking or stretching the duration ofeach object when a presentation cannot be generated, whereasthe ETM [19] uses the interactive primitives to correct in-compatible constraints.

For spatial synchronization, the spatial position of an objectwithin a multimedia presentation can be specified simply interms of its minimum bounding box (MBB), and the spatialdependency between two objects can be expressed using thespatial relations defined in [17]. Moreover, spatial constraintsalso have an implicit temporal dimension that must be takeninto account when modeling a multimedia scenario. So we mustconsider not only static but also dynamic spatial constraints.The problems behind spatial constraints also include complete-ness checking, consistency checking, and constraint relaxation.There exist some important contributions in the literature.For example, there have been several proposals [9]–[11] thatuse the concept of MBB to represent the spatial position ofan object at presentation time, and the approach in [8] alsoprovides the bounding volume for three-dimensional (3-D)objects. However, most of them considered only static spatialconstraints; only the STOG model in [11] supports dynamicspatial constraints. For checking spatial constraints, we needto check inconsistencies between spatial constraints as well asinconsistencies between spatial and temporal constraints. Only[11] proposed a method to check the inconsistency betweenspatial and temporal constraints.

This paper presents the integrated modeling and verificationof multimedia constraints, and relaxation strategies. To ourknowledge, there are no known results reported on checkingthe incompleteness of synchronization constraints, hence noenough constraints to fix layout of presentation. We use a newconstraint analysis model based on directed graphs, whichdiffers from Firefly and ETM, to solve the problems on com-pleteness checking, consistency checking, constraint relaxationand layout generation, interactively or automatically. We firstpropose the concept of the completeness of constraints and di-vide the constraints into different categories, and then introducethe priority number to identify them so as to apply differentpolicies in removing inconsistencies, and use a directed graphto express the constraints among multimedia objects (vertices).Based on those ideas, we develop the algorithms for checkingtemporal and spatial constraints. These algorithms improveefficiency in checking consistency with a simpler model.Moreover, we solve the above problems in a unified theoreticalframework.

TABLE IINTERVAL RELATIONS

This paper is organized as follows. In Section II we ana-lyze the temporal and spatial synchronization constraints ofmultimedia data, and propose a synchronization constraintanalysis model based on directed graphs. Section III addresseshow to eliminate incompleteness and inconsistencies in tem-poral synchronization constraints. We also present strategiesfor relaxing the synchronization constraints. Section IV pro-vides an algorithm to check spatial constraints and generatea spatio–temporal layout from completeness and consistencyconstraints. Section V discusses the method of dynamicchecking for incomplete and inconsistent constraints in userinteractive authoring. Section VI analyzes the efficiency ofalgorithm and compares it with the other previously-proposedsolutions. Finally, Section VII concludes the paper and outlinesfuture research directions.

II. SYNCHRONIZATIONS CONSTRAINTS

A. Temporal Relations

The presentation of a media object usually lasts for a prespec-ified time interval. In order to specify temporal synchroniza-tion, we consider the interval relations between object presenta-tions. Assume that objects and are scheduled in the interval

, , respectively. As shown in Table I, there are13 possible temporal relations between A and B: equals, be-fore, meets, starts, finishes, during, overlaps, and their reverserelations [1].

In order to simplify the representation of temporal relations,we merge some of them, and introduce new notations as follows.

means the presentationinterval of is covered by that of . It is used to specifyconcurrent object presentations instead of , , , .

means the existence of partialoverlap in presenting and . It is used to specify thecross-presentations of objects in the same way as .

means that the presentation of Bis after that of A with delay . It is used to specify thesequential presentations of objects instead of , .

Table II gives the definitions of three constraints.We can easily prove the following theorem by representing

all of the above 13 temporal relations using Cov, Ovl, and Seq.Theorem 1: The constraint set consisting of Cov, Ovl and Seq

is a complete set of temporal relations.If there is a constraint between and , their presentation

intervals have a unique relationship. Fig. 1 illustrates three tem-poral synchronization constraints.

MA AND SHIN: CHECKING CONSISTENCY IN MULTIMEDIA SYNCHRONIZATION CONSTRAINTS 567

TABLE IINEW TEMPORAL RELATIONS

Fig. 1. Diagram of temporal constraints.

When a user authors a multimedia scenario with a constraint-based system, s/he must specify the constraints among the mediaobjects. Actually, the user may have preference to some tem-poral constraints, and in such a case, the system should ensurethem to be satisfied. To facilitate the removal of inconsistenciesin constraints, we divide the constraints into three categories andgive them different priority numbers.1 The constraint with thehighest priority level is viewed as hard constraint, other con-straints can be viewed as soft constraints with default prameters.The priority number for a constraint can be defined by the user.For instance, three levels are given: 1-guaranteed, 2-negotiable,3-negligible, where 1 means higher priority than 2 and 3.

Example 1: Consider a multimedia scenario which firstpresents a logo consisting Logo-Ani (LA) and Logo-Music(LM); then play Audio1 (A1) with Video1 (V1), Text (T1)will be displayed while playing V1; at end, play Exit-Music(EM) with Exit-Ani (EA), V1 may overlap with EM. Sevenconstraints are used to specify the scenario as follows:

We assign the priority number 1 to , , ; 2 to , ,; and 3 to .

B. Spatial Relations

In a multimedia scenario, visual objects, such as video, text,graphic and animation, need to occupy the specific screen spacefor presenting their contents. MBB is a simple way for spec-ifying the space. The MBB defines the display location of anobject by specifying data such as width , height , and thecoordinates of the center of the MBB covering the object.

1One can define more priority levels. For the purpose of illustration, though,three priority levels suffice.

Moreover, to specify the relative depth ordering which reflectsthe obstruction relation among the objects, MBB should providea presentation depth plane . Thus, a MBB is represented bya five tuple . Here we assume that the presentationplane with a larger is nearer the viewpoint than others.

In order to define spatial relations, we introduce the followingnew notations where , .

• means that the presentation space of Acontains that of B. In other words, the presentation spaceof B is inside of that of A.

• means that the presentation space of Acovers that of B.2 In other words, the presentation space ofB is hidden by that of A.

• means the existence of a partial overlapin the presentation spaces of A and B. It is used to specifythe cross-presentation spaces of two objects.

• means that the presentation space of Ais disjoint from that of B.

Suppose that the MBBs of A and B are expressed as, , respectively.

Then, Table III shows the definitions of three constraints byusing MBB ( , , ).Fig. 2 shows the diagrams of these three constraints.

We consider the above four spatial relations as the primitivespatial constraints, since they can express all possible spatialrelations between any two MBBs. Furthermore, we representdynamic spatial constraints as a pair , where is a set ofstatic spatial constraints and is a time interval. It means thatthe spatial constraints specified by are satisfied and remainunchanged within the interval .

C. Model for Analysis of Synchronization Constraints

We first address the problem of analyzing temporal syn-chronization constraints. We propose a Temporal ConstraintsGraph (TCG) model to specify the temporal synchronizationconstraints of a scenario, which is a weighted directed graph.

Definition 1: A TCG is defined as a three tuple, where is the set of vertices rep-

resenting multimedia objects, is the set of edgesspecifying the temporal constraints between objects. There isa directed edge from to if there is a constraint

between them, where is one of three primitive tem-poral constraints. is a mapping fromto a positive integer defining the priority of each constraint.

A directed graph can be stored as a collection of adjacencylists or an adjacency matrix. The adjacency list representation isusually preferred to store sparse graphs for which is muchsmaller than , where denotes the number of edges in thegraph. So, we assume that an input graph is represented in theform of adjacency lists, in which is stored as an elementof the list . There may be more than one constraint be-tween them, but it obviously causes a constraint inconsistency.In such a case, which is the best for an acceptable constraintis decided by comparing these constraints and reporting the in-consistencies to the user. The best one should have the minimum

2This relation is possible although it is meaningless for the spatial layout ofobjects.


Fig. 2. Diagram of spatial constraints.

TABLE IIISPATIAL RELATIONS

priority number, or the one the user interactively chooses. Thedata structure for an edge in an adjacency list is as follows:

type Constraintstruct Edge {Integer ; directed vertex numberConstraint ; constraint typeInteger , ; constraint parameters

Integer ; priority number,

Edge ; pointer to next edge}

Thus, the TCG can be constructed with a collection ofadjacency lists.

Algorithm 1. Preprocessing TCGProcedure PreTCG(Con,TCG,RCon)

Con is the list of constraints, TCG isthe output collection of adjacency lists,RCon is the list of removed constrainsfor each constraint Con doif there exists no in thenappend to as

else if thenmove to RCon;replace by

else redundant constraintsappend to RCon

endif

Fig. 3. Temporal constraints graph.

After preprocessing the TCG, there is at most one constraintfor any pair of vertices.

Fig. 3(a) shows the TCG of the scenario in Example 1.Besides the temporal constraints, we also need to analyze dy-

namic spatial constraints during the presentation. Actually, thepresentation consists of scenes in which the spatial relationsamong objects are not changed. Thus, we also use a directedgraph to model spatial constraints.

Definition 2: Spatial Constraints Graph (SCG) is defined as afour tuple , whereis the set of vertices representing (visual) objects in a scene,

is the set of edges specifying the spatial constrainsbetween objects. There is a directed edge from toif there is a constraint between them, where is oneof four primitive spatial constraints. is amapping from to positive integer defining the priority of eachconstraint. is a mappingfrom to a set of MBBs defining the presentation space of eachobject.

Note that the SCG requests the static spatial constraints for ascene in a multimedia scenario, whereas the TCG does the tem-poral constraints for the multimedia scenario. The data structureof SCG is similar to that of TCG. We can define the dynamicspatial constraints for a multimedia scenario as follows.

Definition 3: Dynamic Spatial Constraints Graph (DSCG)is defined as a sequence

, where is the number of scenes,represents the -th scene specified by spatial

constraints ’s and presented in the interval .In general, for authoring a multimedia title, we first define

the temporal layout of presenting media objects and obtain a se-quence of scenes composed of object presentations, then definethe spatial layout of object presentations in every scene. Finally,the spatio–temporal layout of a scenario is generated.


III. TEMPORAL CONSTRAINT CHECKING

We first discuss the completeness and consistency checkingfor TCG, and then consider spatial constraints checking inSection IV.

A. Completeness Checking

Definition 4: The temporal constraints of a multimedia sce-nario are said to be complete if there is a given or potential (orimplied) constraint or for any pair of ob-jects .

Since an undirected graph is connected if every pair of ver-tices are connected, the concept of undirected TCG can be usedto map the completeness of synchronization constraints.

Definition 5: If an undirected TCG is a connected graph, itis called a Complete TCG (CTCG).

An undirected TCG can easily be obtained from TCG. Weneed to determine if a TCG is a CTCG. If not, the constraints arenot enough to fix the temporal relationships among the verticesin TCG. In general, an undirected graph has some connectedcomponents. An undirected graph is connected if it has exactlyone connected component. We can use Algorithm 2 to find theconnected components of TCG. Note that MAKE-SET cre-ates a new set whose only member is pointed to by , FIND-SET

returns a pointer to the representative of the set containing ,UNION unifies the dynamic sets that contain and intoa new set.

Algorithm 2. CTCG JudgmentProcedure JudgCTCG(Tcg)for each vertexdo MAKE-SET ;

for each edgedo if FIND-SET FIND-SETthen UNION ;FIND-SET (1);

list S; output a connected component;

while dofor any do {

FIND-SET ;list ;

If a TCG has more than one connected component, theconstraints are incomplete. In this case, there is no constraintamong the connected components, so we cannot fix the tem-poral relations among them. Additional constraints should beadded among the vertices of different components. Actually,the user only needs to add constraints if there areconnected components. TCG will become a connected graphafter appending constraints. Fig. 3(b) is the CTCG of Example1 by adding (LA,V1,0) (its priority number W

).

B. Consistency Checking

The consistency checking of synchronization constraints isvery important for producing a feasible presentation. If there isany inconsistency in constraints, the specification of scenariocannot be interpreted into an executable layout. In fact, we havealready removed some redundant constraints while prepro-cessing the TCG. Now, we consider the inconsistency checkingof a CTCG which is considered as a connected, undirectedgraph.

Theorem 2: The temporal layout of presenting media ob-jects can be fixed by constraints among them, where eachobject appears at least in one temporal constraint.

We can prove this theorem by constructing an acyclicconstraint graph (tree).

Inconsistencies are usually due to the redundancy in con-straints. If there are more than edges in a CTCG rep-resenting constraints among objects, there must exist redun-dant constraints. In order to find the best constraints theuser hopes to have, we consider the priority number of each con-straint and then find an acyclic subset that connects allof vertices and whose total weight (priority number)

is minimized. Since is acyclic and connects all of vertices,it must be a spanning tree. So the minimum spanning tree al-gorithm can be used to solve this problem. A solution is to useKruskal’s algorithm [6] as follows.

Algorithm 3. Kruskal’s algorithmProcedure MPST(Tcg,T)

Tcg is a CTCG, T is the outputNULL;

for each vertexdo MAKE-SET ;sort the edges of by nondecreasing

;for each edge

in order by nondecreasingdo if FIND-SET FIND-SETthen

return T

Fig. 4(a) gives the minimum priority number spanning tree(MPST) for the CTCG of Example 1. After finding the minimumspanning tree of CTCG, we can also find some constraintsthat are not selected by if the number of CTCG constraints isgreater than . These constraints are possibly inconsistentwith those in . In order to remove the inconsistency in CTCG,we need to determine if the remaining constraints are accept-able. We now present methods for inconsistency checking andconstraint relaxation.

First, take a constraint in non-decreasing order of , and join with ,then there must be a cycle shown in Fig. 4(b). Assume


Fig. 4. MPST and redundant constraints.

TABLE IVPOSSIBLE PRIORITY NUMBER COMBINATIONS

, where is pro-duced by . According to the MPST algorithm, it is impossiblethat is less than the others’ in . Thus, it is nomore important than others for the user (as Table IV, where

, and the constraints are with3 priority levels). But we have to decide if is acceptable.

From the constraints in except , we can get the con-straint relation between and by using the definitions ofconstraints repeatedly, and Table V shows the formula we used.Note that we should consider the direction of a constraint (anedge of the directed TCG) to find the correct relation.

Thus, the relaxation policy for inconsistent constraints canbe determined by the user once they are found. We just give adefault policy. Given that , , where ,are the start time and end time of , respectively, there are sixcases in total, corresponding to six different relations between

and , as shown in Table VI.

Now, we take to compare with the relationfound, and then decide how to relax the synchronization con-straints in order to obtain a feasible specification.

1) If their constraint types and directions are same, they arecompatible. They can be relaxed according to their prioritynumbers.

a) If , (2,2) or (3,3)),the parameters of are relaxed as follows:

(if necessary).is appended to the CTCG, and remove

the constraint with priority number which is thenearest node or in .

b) If and or(2,1)), then keep the original hard constraints, regard

as an inconsistent constraint, and hence re-move it from the constraint set.

c) If and , the parameters of arerelaxed as follows:

(if necessary).

TABLE VCONSTRAINT RELATION

TABLE VIPOSSIBLE RELATIONS

where is typically taken as .. is appended to the

CTCG, and remove the constraint between and.

2) If their constraint types are same but their direc-tions are different, only Cov constraints are compat-ible, i.e., ,

. The relaxation can be done accordingto their priority numbers.

a) if , we modify the parameters of asfollows: , . is appendedto the CTCG, and remove the constraint with prioritynumber and the nearest node or in cycle .

b) If , then keep the original constraints,regard as an inconsistent constraint, and remove itfrom the constraint set.

3) If their constraint types are different or they have differentdirections and have the same type except Cov, they are con-sidered as incompatible. They cannot be relaxed. If the userdoesn’t select one when their priority numbers are same, is,by default, an inconsistent constraint, and should be removedfrom the constraint set.

After relaxing all redundant constraints, we can obtain thefinal TCG. The final TCG (MPST) of Example 1 is Fig. 4(a)with (A1,T1,2,2) relaxed and moved.

C. Generation of Temporal Layout

In order to generate the temporal layout of a scenario from theCTG, MPST is considered as a directed graph. We can use thebreadth-first traversal to find the unique path from a given vertex

(as the source) to all other vertices. For example, the pathfrom to : , .

Assume that the presentation interval of object is .Once such a path is found, we can get the temporal relationbetween and according to the constraints in this path

, , where and rep-resent the start and end times of presenting , respectively.Thus, we get a list of timed events consisting of the start


Fig. 5. Temporal layout of Example 1.

and end presentation events, denoted by and , respec-tively.

.Then, replace by and sort them in the

temporal order. Note that is the length of :, where

is a start or end event of presenting an object. If thefirst event is scheduled at time 0, take ; then is re-placed in the above list by , and the temporal layout of pre-sentation is as follows: .

Fig. 5 shows the final temporal layout of Example 1 generatedby the above method.

IV. CHECKING SPATIAL CONSTRAINTS

A. Completeness Checking for Spatial Constraints

After obtaining the final temporal layout of a scenario, weneed to specify the spatial constraints among the media objectsin each scene. The temporal layout consists of the presentationintervals in which the specific objects are being presented, so weuse DSCG to represent the spatial constraints of each interval(they are inputed in batch or one-by-one interactively).

Example 2: Consider a multimedia scenario of Example 1again. We assume two objects are appended to the scenario: oneis the background image (B) and the other is the caption (C).Both of them are present for the whole scenario. Assume thatthe screen resolution is 512 512, ,

. The sizes of other visual objectsare , ,

, . Ac-cording to the temporal layout, the scenario can be ex-pressed as ,

.We need to specify the spatial constraints for each scene (de-

noted as ) and assign them specific priority numbers. Forinstance

(B,LA,0,0), S12 (LA,C,0,-220) and ;

, (B,V1,0,0),(V1,C,0,-220) and ;

(B,V1,0,0), (V1, T1,0, 220), (B,T1,0,200) and

;(B,EA,0,0), (EA,C,0,

and .Before generating the final spatial layout for each scene,

we also need to check spatial constraints further. After pre-processing SCG, the completeness checking for the spatial

Fig. 6. Spatial constraints graph.

constraints is similar to that for the temporal constraints. Thatis, the completeness checking of SCG is equivalent to theconnectivity checking of the undirected SCG. If a SCG hasmore than one connected component, the spatial constraintsare incomplete. Additional constraints should be added amongthe vertices of different components. Actually, the user onlyneeds to add constraints if there are connected com-ponents. SCG will become a connected graph after appendingconstraints.

Fig. 6(a) is the Complete SCG (CSCG) of in Example2 by adding (C,T1,0,440) and .

B. Consistency Checking for Spatial Constraints

The consistency checking of spatial constraints is necessaryfor producing a feasible spatial layout. In fact, inconsistenciescome from contradictory spatial relations. We also get resultssimilar to Theorem 2: the spatial layout of presenting mediaobjects (only considering the location ) can be fixed by

spatial constraints among them, where each object ap-pears at least in one constraint. Inconsistencies are usually dueto the redundancy in spatial constraints. As in checking the con-sistency of temporal constraints, we can use the minimum span-ning tree algorithm to find the best constraints the userhopes to have, so that the total weight (priority number) of thesespatial constraints is minimized.

After finding the minimum spanning tree of CSCG[Fig. 6(b)] gives the minimum priority number spanning treeof ), we can find some constraints that are not selectedby the tree if the number of CSCG constraints is greater than

. Such constraints are possibly inconsistent with thosein . In order to remove the inconsistency in CSCG, we needto determine if the remaining constraints are acceptable andhow to relax it, so that it may become consistent with the

constraints. The procedure is similar to that of checkingredundant temporal constraints:

1) Take a constraint in nondecreasingorder of priority numbers, and join to . Then,there must be a cycle that contains produced by .Assume that is an implied relation between and in

, and .2) Compare with , then decide how to relax the spatialconstraints.

a) If ( , (2,2) or (3,3)),the parameters of are relaxed as follows:

; .


is appended to the CSCG, and remove theconstraint with priority number which is the nearestto node or in .

b) If and or(2,1)), then keep the original hard constraints, regard

as an inconsistent constraint, and hence re-move it from the constraint set.

c) If and , the parameters of arerelaxed as follows: ;

. where is typicallytaken as .

. is appended to the CSCG,and remove the constraint between and .

Note that the constraint type of will possibly be changedbecause of the adjustment of the location in Case 2(a) and 2(c).The constraint direction of remains unchanged. The finalSCG (MPST) of is given in Fig. 6(b) (S33 is viewed asan inconsistent constraint).

C. Generation of Spatial Layout

From the selected constraints, we can fix up the finalpresentation location of each object. To generate the cor-rect spatial layout, we also consider the relative ordering in thepresentation plane of intersecting spaces.

We view the Complete SCG as a directed graph again. A di-rected edge in the SCG indicates that the presentationplane of will cover that of , i.e., . In order to findthe correct depth ordering of presentation, we process CSCG asfollows.

1) Check if there exists a directed cycle by the depth-firstsearch (DFS) algorithm [6]. If there exists one, just delete oneconstraint which is not in MPST from the cycle to eliminateunordered presentation plane relations. Thus, we can get anacyclic directed graph , and MPST is also in the graph.2) Find the topological sort of . It is a linear or-

dering of all vertices such that if contains a directed edge, then appears before in the ordering. The

topological sort algorithm is presented in [6], and its runningtime is .3) Take an object vertex without any incoming edge (usually

the background image with the minimum as the source),and generate an ordering of presentation planes according tothe topological sort found. The topological sort of isshown as in Fig. 7. Note that the ordering is not unique.4) Check the generated ordering. If the ordering is not satis-

factory, the user can modify constraints or append new con-straints to CSCG, and repeat the above procedure.After obtaining a satisfactory ordering of object presentation

planes in a scene, we can generate the final spatial layout bycombining the the location information of objects.

When the MBB of background image B is 256, 256, 512,512, 0 , the final spatial layout of Example 2 is generated asshown in Fig. 8.

V. SUPPORT FOR INTERACTIVE AUTHORING

The user can interactively edit the specification of a scenarioboth on authoring temporal and spatial relations, so the approach

Fig. 7. Topological sort of SCG .

Fig. 8. Spatial layout of Example 2.

should support the interactive authoring. For authoring temporalrelations among the media objects, possible editing operationsare as follows.

Add an object : addition of a new media objectalso needs a constraint between and an old object (vertex).If there are more than one such constraint, we select one withthe minimum priority number first and add it to the CTG. At thesame time, its addition to the old MPST forms the new MPSTof modified graph. Then, consistency checking for additionalconstraints on will be done.

Remove an object : deletion of a media objectalso causes inefficient constraints among and other objects(vertices). If there is only one such constraint in the old MPST( is a leaf node), we just delete it to form a new MPST. Other-wise ( isn’t a leaf node), we need to delete all constraints on

from the old TCG, and find a new CTCG and a new MPST.Moreover, consistency checking will be done again.

Add a constraint : addition of a new constraintmay cause MPST to change. A cycle will be formed after adding

to MPST. If the priority number of is less than that of aconstraint, denoted as , with the maximum priority number,then we can delete and add in the old MPST to form thenew MPST. Otherwise, the consistency checking for needs tobe done.

Remove a constraint : deletion of a constraintmay cause incomplete constraints. Thus, we need to delete thisconstraint from the old TCG, and find a new CTCG and a newMPST. Moreover, consistency checking is also done again.

Modify a constraint : modification of a constraintcan be divided into two cases. In the first case where C.pnum isunchanged, MPST will remain unchanged and the other param-eters of may be modified if is in MPST. In the second case,C.pnum is changed. We need to reconstruct the MPST fromCTCG with the modified .

To author the spatial relations among media objects, wesimply include “Add/Remove/Modify a spatial constraint.” Theoperations can be processed similarly to the case of editingtemporal constraints.


VI. ANALYSIS AND COMPARISON

A. Efficiency of Algorithm

Our approach is divided into two steps. The first is checkingthe temporal constraints. Suppose there are multimedia ob-jects, and temporal constraints among them. The checkingprocedure consists of the following.

• Preprocessing TCG which generates the temporal con-straint graph taking time.

• JudgeCTCG which is used to judge if the TCG is CTCG,with a total of time.

• MPST(CTCG) which takes time to find theminimum priority number spanning tree.

• Removing inconsistency which takestime.

• Generating temporal layout which includes the tree tra-versal and sorting timed events, taking time.

Generally speaking, and is close to .So, the total running time of checking temporal constraint is

.The second step is to check spatial constraints. Assume that

the scenario consists of scenes, and there are objectsand constraints among them. For each scene, we need to usethe following algorithms.

• Preprocessing SCG which generates the spatial con-straint graph taking time.

• JudgeCSCG which is used to judge if the SCG is CSCG,taking time.

• MPST(CSCG) which takes time to find theminimum priority number spanning tree.

• Removing inconsistent spatial constraints which takestime.

• Generating spatial location includes the tree traversal,taking time.

• Sorting presentation planes includes the topological sortalgorithm, taking time.

Because and is close to , for a sce-nario, the total running time of checking spatial constraint is

or .

B. Comparison With Related Work

As mentioned in Section I, there are many multimedia syn-chronization models. However, most of them do not provideconsistency checking. Compared to several main consistencychecking models, such as OCPN [12], [18], Firefly [5], TCSP[2], ETM [19], Time Automata [7] and Temporal Logic[13]–[15], our method has the following advantages.

1) Integrated Modeling and Verification of Multimedia Con-straints: We proposed the concept of completeness of synchro-nization constraints, which had previously been overlooked. Forconvenience we divide the constraints into different categoriesand introduce the priority number to identify them, so that wecan adopt different policies in removing inconsistencies. Byusing a directed graph to express the temporal and spatial con-straints among multimedia objects, we solve the problems oncompleteness checking, consistency checking, constraint relax-ation and layout generation in a unified theoretical framework.

2) A Simple Method to Develop the Analysis Tools to Sup-port a Constraint-Based System: Our method is based ongraph theory which turns the problems in constraint-basedsystems into simple solved problems, i.e., the completenessof constraints is mapped to connectivity of TCG (SCG), andfinding best constraints is mapped to a spanning tree of TCG(SCG). OCPN gives the semantic specification of temporalconstraints, TCSP, Temporal logic and Time Automata alsoefficiently specify temporal synchronization. Although theirproof theory may be used to check inconsistency, it is difficultto develop automatic checking tools for practical multimediaauthoring. Moreover, it is difficult to use these methods tocheck spatial constraints.

3) More Efficient Solution: Our constraint analysis model isdifferent from Firefly and ETM although all of them are basedon graphs. Firefly and ETM use graphs to specify the temporalrelationship among the start and end events (in presenting ob-jects) by using vertices to represent events and edges to labelthe relations between the events. In contrast, our model usesvertices to represent objects, and edges to label the constraintsbetween objects. The complexity is reduced because its reduc-tion of the number of vertices by one-half. Moreover, the under-lying structure of a multimedia scenario is easily manipulatedin user’s interactive authoring with this model. Firefly providesa scheduling algorithm to produce a presentation schedule; if noschedule is derivable from the specification, the constraints aresaid to be infeasible (including inconsistency). ETM uses a sim-ilar idea to check the consistency of temporal constraints withmore flexibility. However, they can’t handle the inconsistencychecking of spatial constraints. By using our method, the incon-sistency checking for temporal constraints will be done beforegenerating the feasible temporal layout of presentation, and ourmethod can also be used to check spatial constraints. So, ourmethod improves the efficiency of checking consistency thanksto a simpler model than others.

4) Flexible Implementation: Our method solves the aboveproblems in both interactive and batch authoring systems,especially for the multi-user collaborative authoring mode.In an interactive authoring system, the user can interactivelyselect the constraint relaxing policy to facilitate consistencychecking, constraint relaxation and layout generation. In abatch authoring system, the default processing will solve theseproblems automatically.

VII. CONCLUSIONS

This paper proposes a new method of analyzing multimediasynchronization constraints based on graphs and classificationof constraints, which is essential to the development of efficientsystem support tools for constraint-based authoring systems.

This method provides a unified theoretical framework tosolve the problems in constraint-based multimedia authoringsystems, such as completeness checking, consistency checking,constraint relaxation and automatic spatio–temporal layoutgeneration. A complete set of algorithms for analyzing tem-poral and spatial constraints are developed. We also discuss theeffects of user’s interactive authoring. Compared to previous


methods, the main advantages of our method are simple, effi-cient, and easy to implement and can be used by nontechnicalusers. Future work includes the development of more sophis-ticated consistency checking techniques considering user’sinteractivity and transformable presentation spaces, and theoptimal solution to the presentation under soft constraints aswell as the issue related to the quality of presentation.

REFERENCES

[1] J. Allen, “Maintaining knowledge about temporal interval,” Commun.ACM, vol. 26, no. 11, pp. 832–843, 1983.

[2] A. F. Ates, M. Bilgic, S. Saito, and B. Sarikaya, “Using timed CSPfor specification, verification and simulation of multimedia synchroniza-tion,” IEEE J. Select. Areas Commun., vol. 14, pp. 126–137, Jan. 1996.

[3] E. Bertino and E. Ferrari, “Temporal synchronization models for mul-timedia data,” IEEE Trans. Knowl. Data Eng., vol. 10, pp. 612–631,July-Aug. 1998.

[4] G. Blakowski and R. Steinmetz, “A media synchronization survey: Ref-erence model, specification, and case studies,” IEEE J. Select. AreasCommun., vol. 14, pp. 5–35, Jan. 1996.

[5] M. C. Buchanan and P. T. Zellweger, “Specifying temporal behavior inhypermedia documents,” in Proc. ACM Conf. Hypertext, Milan, Italy,1992, pp. 262–271.

[6] T. H. Cormen, C. E. Leiserson, and R. L. Rivest, Introduction to Algo-rithms. Cambridge, MA: The MIT Press, 1990.

[7] J. P. Courtiat and R. C. De Oliveira, “Proving temporal consistency in anew multimedia synchronization model,” in Proc. ACM Multimedia96,Boston, MA, 1996, pp. 141–152.

[8] Y. F. Day et al., “Object-oriented conceptual modeling of video data,”in Proc. 11th Int. Conf. Data Engineering, Taiwan, R.O.C., 1995, pp.401–408.

[9] A. Ghafoor and Y. F. Day, “Object-oriented modeling and querying ofmultimedia data,” in Proc. First Int. Workshop on Multimedia Informa-tion Systems, Arlington, VA, Sept. 1995, pp. 111–119.

[10] P. Hoepner, “Synchronizing the presentation of multimedia objects,”Comput. Commun., vol. 15, no. 9, pp. 557–564, 1992.

[11] Y.-M. Kwon, E. Ferrari, and E. Bertino, “Modeling spatial-temporal con-straints for multimedia objects,” Data Knowl. Eng., vol. 30, pp. 217–238,1999.

[12] T. D. C. Little and A. Ghafoor, “Synchronization and storage modelsfor multimedia objects,” IEEE J. Select. Areas Commun., vol. 8, pp.413–427, Apr. 1990.

[13] H. Ma and S. Liu, “Multimedia data modeling based on temporal logicand XYZ system,” J. Comput. Sci. Technol., vol. 14, no. 2, pp. 188–193,1999.

[14] H. Ma and X. Tang, “The design and implementation of multimedia au-thoring language,” J. Software, vol. 9, no. 12, pp. 889–893, 1998.

[15] H. Ma, “The specification and verification of multimedia synchroniza-tions in duration calculus,” J. Comput. Sci. Technol., vol. 18, no. 2, pp.172–180, 2003.

[16] I. Mirbel, B. Pernici, and M. Vazirgianis, “Temporal integrity constraintsin interactive multimedia documents,” in Proc. IEEE ICMCS’99, vol. 2,Florence, Italy, June 1999, pp. 867–871.

[17] D. Papadias et al., “Topological relations in the world of minimumbounding rectangles: A study with R-trees,” in Proc. ACM SIGMOD95,San Jose, CA, 1995, pp. 92–103.

[18] N. U. Qazi, M. Woo, and A. Ghafoor, “A synchronization and commu-nication model for distributed multimedia objects,” in Proc. ACM Mul-timedia93, 1993, pp. 147–155.

[19] J. Song et al., “Interactive authoring of multimedia documents in aconstrain-based authoring system,” ACM Multimedia Syst., vol. 7, pp.424–437, 1999.

Huadong Ma (M’99) received the B.S. degreein mathematics from Henan Normal University,China, in 1984, the M.S. degree in computer sciencefrom Shenyang Institute of Computing Technology,Chinese Academy of Science (CAS) in 1990 and thePh.D. degree in computer science from Institute ofComputing Technology, CAS, in 1995,

He is a Professor with the School of ComputerScience & Technology, Beijing University of Postsand Telecommunications, China. He visited TheUnited Nations University/International Institute for

Software Technology (UNU/IIST) as a Research Fellow in 1998 and 1999.From 1999 to 2000, he held a visiting position in the Real-Time ComputingLaboratory, Department of Electrical Engineering and Computer Science,The University of Michigan, Ann Arbor. His current research focuses onmultimedia, networking, e-commerce and computer graphics. He has publishedover 60 papers and three books in these fields.

Dr. Ma is a member of the ACM.

Kang G. Shin (S’75–M’78–SM’83–F’92) receivedthe B.S. degree in electronics engineering fromSeoul National University, Seoul, Korea, in 1970,and both the M.S. and Ph.D. degrees in electricalengineering from Cornell University, Ithaca, NY, in1976 and 1978, respectively.

From 1978 to 1982, he was on the faculty of Rens-selaer Polytechnic Institute, Troy, NY. Currently,he is the Kevin and Nancy O’Connor Professor ofComputer Science and Founding Director of theReal-Time Computing Laboratory, Department of

Electrical Engineering and Computer Science, The University of Michigan,Ann Arbor. His current research focuses on QoS-sensitive networking andcomputing as well as on embedded real-time OS, middleware and applications,all with emphasis on timeliness and dependability. He has supervised the com-pletion of 49 Ph.D. theses, and authored/coauthored over 600 technical papersand numerous book chapters in the areas of distributed real-time computingand control, computer networking, fault-tolerant computing, and intelligentmanufacturing. He has also held visiting positions at the U.S. Airforce FlightDynamics Laboratory, AT&T Bell Laboratories, EECS Department at the Uni-versity of California, Berkeley, the International Computer Science Institute,Berkeley, IBM T. J. Watson Research Center, Yorktown Heights, NY, SoftwareEngineering Institute at Carnegie Mellon University, Pittsburgh, PA, and HPResearch Laboratories. He also chaired the Computer Science and EngineeringDivision, EECS Department, The University of Michigan, for three yearsbeginning January 1991. He is an Area Editor of the International Journal ofTime-Critical Computing Systems, Computer Networks, and ACM Transactionson Embedded Systems.

Dr. Shin is a Fellow of IEEE and ACM, and a member of the KoreanAcademy of Engineering. He was the General Chair of the 2000 IEEEReal-Time Technology and Applications Symposium, the Program Chair ofthe 1986 IEEE Real-Time Systems Symposium (RTSS), the General Chair ofthe 1987 RTSS, the Guest Editor of the August 1987 IEEE TRANSACTIONS

ON COMPUTERS special issue on Real-Time Systems, a Program Co-Chairfor the 1992 International Conference on Parallel Processing, and served onnumerous technical program committees. He also chaired the IEEE TechnicalCommittee on Real-Time Systems during 1991–1993, was an Editor of theIEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED COMPUTING. Hehas received numerous prestigious awards, including an Outstanding IEEETRANSACTIONS ON AUTOMATIC CONTROL Paper Award (1987), the IEEECommunications Society William R. Bennet Paper Prize (2003), the ResearchExcellence Award (1989), and the Stephen Attwood Award (2004) from theUniversity of Michigan.

Documents

Checking Consistency in Multimedia Synchronization Constraints