Wolf-Tilo Balke Preference-Driven Personalization for ... · on what MPEG-7/21 can actually express in user pref-erences. The User Interaction Tools (UIT) oﬀer several possibil-ities

Multimedia Systems manuscript No.(will be inserted by the editor)

Benjamin Kohncke · Wolf-Tilo Balke

Preference-Driven Personalization for Flexible DigitalItem Adaptation

Abstract The delivery of multimedia content often needsthe adaptation of the content in order to satisfy userconstraints. With the Digital Item Adaptation part, theMPEG-21 standard already defines a useful frame-workto handle this task. However, in modern service-orientedarchitectures the functionality of adaptation is split overseveral services. Hence, the central instantiation of asuitable service chain needs to tackle a complex multi-objective optimization problem. In this problem betweencontent choice and possible adaptations the current pref-erence model in the MPEG-7/21 standard still lacks ex-pressiveness. In the course of this paper we demonstratethis shortcoming and how the integration of more power-ful models can ease the instantiation problem. Further-more we explain how to efficiently evaluate preferencetrade-offs by evaluating skyline queries as currently in-vestigated in the field of information systems. As a run-ning example we use preference-based content adapta-tion in a typical media streaming application with Webservices as basic modules. The contribution of our frame-work is to enable a central coordinator to instantiate anexecutable service composition chain by integrating allneeded Web services to adapt the multimedia content inthe best possible fashion in the sense of Pareto optimal-ity.

Keywords Multimedia Systems · Digital Item Adap-tation · Personalization · Skyline Queries · PreferenceModeling

This work was supported in part by the German ResearchFoundation (DFG) within the Emmy-Noether Program ofExellence.Wolf-Tilo BalkeL3S Research Center at the University of Hannover, Han-nover, GermanyTel.: +49-511-762-17712Fax: +49-511-762-17779E-mail: [email protected]

Benjamin KohnckeL3S Research Center at the University of Hannover, Han-nover, GermanyE-mail: [email protected]

1 Introduction

Delivering multimedia content over a plethora of de-vices in a personalized fashion puts great demands onthe selection and adaptation of content [24]. There aretwo basic approaches: either a server side adaptation,where all adapted versions of the appropriate contentare then broadcasted in a stream, or a client side adap-tation where powerful decoders have to be available, toperform an on-the-fly adaptation. Building on service-oriented adaptation our approach goes beyond these ba-sic models and provides an individual service chain foreach user where the adaptation is done on-the-fly. For ev-ery individual user a specific workflow has to be createdrespecting the individual user’s content preferences, gen-eral content semantics and network constraints, as wellas terminal capabilities.This is because today, instead of building complex mono-lithic systems the flexible composition of services intosuitable workflows for media handling is an importantgoal for system integration and reusability of compo-nents (for an overview see [17]). Such workflows have tobe planned and their execution closely monitored, some-times needing the seamless integration of replacementsfor failed services. To facilitate this, the instantiation ofthe service chain needs to solve a multi-objective opti-mization problem considering possible services (as havebeen discovered, cf. [1]), the content currently availableand the author’s, user’s and client device’s constraintson the adaptation.After having addressed the basic problem in [11], in thispaper we discuss how to adapt content in a personal-ized fashion by integrating MPEG-7/21 metadata with acomplex preference framework, namely a model on qual-itative partial order preferences (cf. [6], [9]) and showhow to evaluate the complex result meshes given by therespective product order under the concept of Pareto op-timality.The benefit is twofold: on one hand integrating a morepowerful model enables higher expressiveness of user pref-erences; on the other hand composition engines and ser-

2 Benjamin Kohncke, Wolf-Tilo Balke

vices along the workflow chain can directly use the aug-mented metadata for choosing adaptation strategies moreintelligently. Every service adapts the metadata accord-ing to the transformation of the content it performedand removes obsolete preference information. This alsoenables a more efficient matching of constraints for thoseservices later in a service composition chain.As a running example application throughout this paperwe will use media streaming that flexibly adapts somemultimedia content to the terminal capabilities of theuser. Thus, users get the best possible quality with re-spect to their respective terminal capabilities. Our proto-typical implementation uses Web services as basic mod-ules to build multimedia applications. The description ofthe complex data types is provided by an MPEG-7/21description attached to the media data. Each individualWeb service evaluates the MPEG-7/21 description andadapts the multimedia material to the special needs ofthe user and the client device.To coordinate the composition we currently use a ded-icated central Web service instance that takes over re-sponsibility for selecting the best available content anda suitable workflow by means of the technical profile anduser preferences [1]. This service instantiation and mon-itoring service (SIAM) does also monitor the workflow,but it does not in detail decide in what way each Webservice should adapt the multimedia content. Every in-dividual Web service (e.g., merging or transcoding ser-vices) can use its own heuristics about how to adaptthe selected multimedia content best with respect to thespecified target output for the end-user. In addition tothe purely technical information, the services also use im-portant description schemes from the MPEG-7/21 stan-dard like e.g., transcoding hints [23].The rest of the paper is structured as follows: In sec-tion 2 we will give an overview about the user specificmetadata descriptions offered by MPEG-7/21. A shortsummary about what has already been done in the fieldof personalized media adaptation is given in the relatedwork part in section 3. Then we introduce a more expres-sive preference model in section 4 and present the basicarchitecture of our prototypical system. In section 5 wedescribe how to actually evaluate the complex trade-offmesh of a Pareto preference graph and the prototypicalimplementation of a media streaming test case is dis-cussed in section 6. Finally, we close the paper with ourconclusions and some future research directions.

2 Metadata Descriptions in MPEG-7/21

The MPEG-7/21 Usage Environment is a part of theMPEG-21 Digital Item Adaptation (DIA) architecture(Part 7, ISO/IEC 21000-7) and offers several opportu-nities to define user specific information. It includes theUser Interaction Tools from MPEG-7 [23] and also thedescription elements for the device capabilities of the

user (in the Terminal Capabilities description scheme).Figure 1 shows the typical interaction of a content adap-tation engine with a media database and the end user,respectively his/her client device. Media data generally

Fig. 1 General Digital Item Adaptation

is retrieved by matching metadata descriptions. To per-form a content selection the personalized adaptation de-cision engine matches the resource descriptions from ametadata directory and the user-provided information.This information consists of the actual user query, moregeneral content preferences of the user, and technicalconstraints that provide data about the technical deviceof the user. After determining the optimal adaptation,the decision is handed on to the adaptation engine. Theadaptation engine then retrieves the media resource fromthe media database, adapts it accordingly and deliversthe adapted resource to the user. Basically an adaptationworks as follows:

1. The decision engine gets a request with content pref-erences and technical constraints

2. If the requested media is available and all constraintsare satisfied the respective multimedia data is di-rectly delivered

3. If the requested media is available, but some con-straints are not yet satisfied

3.1. If the multimedia data can be adapted to meet allconstraints, adapt media accordingly and deliver

3.2. If not, choose the highest ranking preference(s)to adapt to other formats, resolutions, etc. or tochoose new content

However, this adaptation scheme cannot work with morecomplex trade-offs, and neither can it decide intelligentlywhen to choose alternative adaptation options that (pos-sibly if taken together) would make more sense than sim-ply the highest rated preference. Let us take a closer lookon what MPEG-7/21 can actually express in user pref-erences.The User Interaction Tools (UIT) offer several possibil-ities to describe content preferences of the user. In con-trast to the other description schemes defined in MPEG-7 the UIT are bound to the user and not to the multi-media material: the User Preferences are used to ex-press specific preferences of the user in the selection of

Preference-Driven Personalization for Flexible Digital Item Adaptation 3

multimedia material and consist of several descriptionschemes:

– A set of Filtering and Search Preferences describesindividual user wishes with respect to the filteringand searching of multimedia data. Each element isdecomposed into:– Filtering and Search Preference elements on a lower

level to allow for preference hierarchies.– Classification Preference elements to describe pref-

erences regarding to attributes, such as language,production format or country of origin.

– Creation Preference elements to describe the pre-ferred creation of content, such as title, creator orcreation date.

– Source Preference elements where preferred repos-itories to retrieve multimedia material from (e.g.,media servers or digital libraries) can be defined.

– Preference Condition elements where the user canconstrain the applicability of specific filtering andsearch preferences (for example time and date).

– A set of Browsing Preferences elements describe theuser’s wishes regarding multimedia content naviga-tion and browsing. A typical example is a SummaryPreference describing the preferred type of contentsummaries.

If the user got more than one preference, a weighting fac-tor, called Preference Value, can be specified to expressthe relative importance of a preference. The preferencevalue is a simple numerical value ranging from -100 to100. The value indicates the degree of users preference ornon-preference. The zero value indicates that the user isneutral in terms of preference versus dislike. A default,positive, value of 10 corresponds to a nominal preference.By choosing a negative weighting the user can expressnegative preferences or dislikes.

Example: The following XML code snippet shows acomplex preference in MPEG-21 notation for user ’Kim’,who prefers ’Action’ movies starring ’Matt Damon’ overthe actors ’Arnold Schwarzenegger’ and ’Brad Pitt’ andon the whole movies that are in German language overmovies in English language using appropriate Filteringand Search Preference elements.

<UserPreference><UserIdentifier userName="Kim"/><UsagePreference>

<FilteringAndSearchPreferences><ClassificationPreference>

<Genre>Action</Genre><Language preferenceValue="90">german</Language><Language preferenceValue="70">english</Language>

</ClassificationPreference><CreationPreference>

<Actor preferenceValue="63">Damon</Actor><Actor preferenceValue="50">Schwarzenegger</Actor><Actor preferenceValue="27">Pitt</Actor>

</CreationPreference></FilteringAndSearchPreference>

</UsagePreference></UserPreference>

To decide between preferences all the preference val-ues within each attribute are compared separately: thehigher the value, the more important the respective pref-erence. However, the MPEG-7/21 standard just definesthe syntax and semantics of the user preference descrip-tion scheme, but not the extraction method of the pref-erence value, cf. [12]. That means all users have to as-sign preference values manually. But can semanticallyincomparable attributes (like preferences on actors andlanguage settings) be compared in a quantitative way?It hardly makes sense to state something like: ’I preferMatt Damon movies to movies in German’. Moreover,for the user it is entirely unintuitive, what an individualpreference value (like 63 or 50) actually means. Other au-thors, e.g. [27], have proposed to use even more complexutility functions, but after all still rely on quite unintu-itive quantitative preference frameworks.In the example above we have two preferences on differ-ent attributes: language and actors, but the attributesare basically incomparable. Thus, some combinations formedia objects become also incomparable and cannot beranked in a total order. This characteristic leads directlyto the concept of Pareto optimality: the Pareto set (oftenalso called efficient frontier) consist of all non-dominatedobjects, i.e. for each object no other object in the sethas better or at least equal attribute values with re-spect to all attributes. Consider for instance an EnglishMatt Damon movie and a German Arnold Schwarzeneg-ger movie. Both are incomparable, because one is bet-ter with respect to the language preference, whereas theother is better with respect to the actor preference. How-ever, both options dominate an English Brad Pitt movie,which accordingly would not be part of the Pareto set.The use of the Pareto semantics is also advocated in [14]providing a decision taking framework where hard- andsoft-constraints are represented as variables as input forthe optimization problem.A simple matching of preference values rarely leads toan effective trade-off management. This is because pref-erences generally distinguish between hard and soft con-straints. Hard constraints have to be adhered to, no mat-ter how (un-)important the respective preference is. Con-sider transcoding hints, where the original author of themultimedia material can define how the properties of themultimedia content can be changed without compromis-ing the content’s semantics. For instance, it could bestated that the resolution of a movie can only be reducedup to 50% of the original resolution. A further reductionsimply does not make sense; even if it is exactly the con-tent a user requested by expressing content preferenceswith high preference values. On the other hand a usermight express a preference for best possible resolution.Such a preference can be considered as a soft constraintthat can be relaxed, if necessary.Generally speaking two hard boundaries always constrainservice compositions (cf. figure 2): the device capabil-ities form upper boundaries for the capabilities of the


terminal, like for example the maximum possible reso-lution. The transcoding hints form lower boundaries forthe quality of multimedia material, like for example theminimum possible resolution. If the device capabilitiesand transcoding hints do not overlap a sensible mediaadaptation without changing the modality is not possi-ble (cf. figure 2).

Fig. 2 Boundaries for Media Adaptation

Basically for adaptation of the multimedia materialour SIAM service uses the InfoPyramid approach [13] tooffer multimedia content in alternative fidelities and/ormodalities. In case of impossible adaptations the user’scontent preferences are used to choose alternative mediaand thus realize a truly cooperative retrieval behavior.

3 Related Work

The basic usefulness of the user preferences descriptionscheme for automatic content adaptation has alreadybeen shown to some degree by a variety of approaches.[25] provides an useful overview of current digital itemadaptation techniques in MPEG-21 and how they canbe utilized by multimedia applications. In particular, theauthors give detailed insights about usage environmentdescriptions which includes, among other things, termi-nal capabilities and network characteristics. Furthermoreongoing standardization activities related to DIA, as wellas several emerging research topics and open issues arediscussed.The semantic information a multimedia document con-veys can be presented within different levels of abstrac-tion. Recently [19] described how to include semanticdescriptions using semantic Web technology. But sincetoday semantic Web languages still lack the structuraladvantages of the XML-based approach and current mul-timedia document annotation is already given by otherlegacy standards, a combining of the existing standardsseems to be the most promising path for multimedia doc-ument description. MPEG-7/21 as one of the interna-tional key standards in this area is defined by a XMLschema and currently no commonly accepted mappingexists from the XML schema definitions to RDF or OWL[19]. Although a further monitoring of the development

of semantic Web media descriptions is necessary, cur-rently we still rely on XML annotations.In terms of adaptation frameworks [20] also uses the sim-ple XML model to design a video personalization andsummarization system in heterogeneous usage environ-ments. The design framework results in a three-tier ar-chitecture of server, middleware and client. Personaliza-tion and adaptation engines, which select, adapt and de-liver summarized media files to the user, are included inthe middleware layer. For each decision if a video shotshould be included in the personalized video summaryor not, the preference scores from the MPEG-7/21 stan-dard are taken. However, unlike in our work presentedhere in all cases these preferences are static and the ap-propriate adaptation is always assumed to be possible.In contrast, the work in [21] goes beyond mere staticadaptation and even provides some novel ontology-basedmethodologies to open up the MPEG-7/21 usage envi-ronment for enriching user preferences by more complexsemantics as expressed by domain ontologies. In doing so,the approach supports the complete functionality offeredby the MPEG-7 semantic description scheme for multi-media content descriptions and respects all the MPEG-7/21 conventions. It is based on the Web ontology lan-guage OWL and has been prototypically implemented inthe DS-MIRF framework. Moreover, in [22] the systemis extended by a Query language for MPEG-7 descrip-tions. Allowing the integration of domain knowledge intosemantic MPEG-7 metadata an automatic relaxation ofover-specified constraints is thus enabled. However, theactual integration of specific user preferences is not yetsupported in this framework and the task of personaliza-tion still remains a challenging issue.In a similar fashion [15] explores diverse algorithms andmethodologies for an ontology-based approach, but herethe focus is on exchanging knowledge between users. Thesemantic relations in the ontology are defined in MPEG-7 and can be visualized using RDF. An agglomerativeclustering approach is used to extract preferences basedon user history and a fuzzified version of this knowledgecan be exchanged.To exploit also specific user preferences and therefore im-prove the user’s interaction experience and understand-ing of the adaptation process the integration of a moreexpressive preference model is needed. An important stepto derive and maintain user preferences in a meaningfulprofile is the elicitation step. There exists a large bodyof work in user modeling (see e.g., [10], [5] for a basicoverview) relying on different techniques to derive a pro-file. Especially suitable group profiles as compromises ora basis for multicast techniques may be helpful in solv-ing some of the demanding QoS issues involved in mediastreaming. For instance, in [12] a use case of an electronicprogram guide is presented, where the user’s interactivebehavior with video programs is monitored, and the userpreferences are updated accordingly. Here, the relativeimportance of different preferences can to some degree


even be generated automatically.In terms of the service-oriented processing of multime-dia data the MPEG-21 standard also introduces networkquality of service (QoS) tools. For choosing adequate net-work paths the usefulness of integrating more complexuser preferences has already been recognized and usu-ally typical non-functional statistics like average serviceavailability or network latencies are already integratedin the decision, e.g. for telephony [8]. However, the ac-tual real-time measurement of statistical information todecide for best service paths is still difficult and beyondthe scope of this paper (see e.g., [7] or [16]).

4 A More Expressive Preference Model

The existence of multiple and often conflicting user pref-erences demands an efficient framework to deal with themin a fair and meaningful way. The need of an effectivetrade-off management with complex user preferences hasalready been considered in other communities like e.g.,in databases and information systems. Here, recent workin [6] and [9] considers preferences in a qualitative wayas partial orders of preferred values that can be relaxedif the need arises. To combine multiple preferences andderive a fair compromise usually the notion of Paretooptimality is used.In the case of content selection and adaptation the re-sults are all possible solutions that are not dominated byother solutions with respect to all attributes specified inthe media request. That means if all preferences on differ-ent attributes are considered to be of equal importance,the sub-optimal solutions are automatically removed andout of the remaining pool of possible solutions an adapta-tion service can pick a suitable instantiation. Of course,if no fair relaxation scheme is desired, also more discrim-inating combination methods (e.g. the ordering on theattributes in preference values in MPEG-7/21) can beused on qualitative partial order preferences.

mpeg

Huffyuv

divx

Indeo

Willis

di Caprio

Schwarzenegger

Damon

Pitt

Fig. 3 Explicit Preference Graphs

Example (cont.): Consider our user Kim owns aPDA and again requests some media for streaming, butthis time explicitly expresses two complex preferences:one about preferred movie actors and one about pre-ferred codecs available of her client device (as defined by

the terminal capabilities). The corresponding preferencegraphs might look as shown in figure 3. Our frameworkextends the MPEG-21 notation to express these prefer-ences in XML, e.g., the first preference, like follows:


<Preference><EXP att="actor">

<EXPSet><Value val1="Damon" val2="Willis"><Value val1="Damon" val2="Schwarzenegger"><Value val1="Willis" val2="di Caprio"><Value val1="Schwarzenegger" val2="di Caprio"><Value val1="Schwarzenegger" val2="Pitt">

</EXPSet></EXP>

</Preference></UsagePreference>

</UserPreference>

To combine several preferences the combination se-mantics has to be stated. For our example let us assumea fair relaxation scheme between the two preferences.Figure 4 shows the first three layers of the product pref-erence graph following the Pareto semantics. The graphis already quite complex for combining only the two pref-erences in figure 3. Please note that due to the qualitativenature of the preferences some combinations are incom-parable: if the best choice (a Matt Damon movie in divxformat) should not be available or adaptable (e.g., miss-ing a suitable transcoding service), the adaptation deci-sion can explore several other options that are all equallypreferable such as a Matt Damon movie in mpeg formator a Bruce Willis movie in divx format.

(Willis, divx)

(di Caprio, divx)

(Schwarzenegger, divx)

(Damon, divx)

(Pitt, divx)

(Damon, mpeg)

(Schwarzenegger,mpeg)...Fig. 4 Pareto Preference Graph

To express such Pareto preferences in XML we allowqualifying the combination of explicit preferences follow-ing the respective preference algebra (see [9] for detailson the evaluation of complex constraints):


<Preference><Pareto>

<EXP att="actor"><EXPSet>

<Value val1="Damon" val2="Willis"><Value val1="Damon" val2="Schwarzenegger"><Value val1="Willis" val2="di Caprio"><Value val1="Schwarzenegger" val2="di Caprio"><Value val1="Schwarzenegger" val2="Pitt">

</EXPSet>


</EXP><EXP att="codec">

<EXPSet><Value val1="divx" val2="mpeg"><Value val1="mpeg" val2="Huffyuv"><Value val1="mpeg" val2="Indeo">

</EXPSet></EXP>

</Pareto></Preference>

</UsagePreference></UserPreference>

For evaluating such an advanced trade-off manage-ment in a service-oriented environment several compo-nents are needed. Figure 5 shows the basic architecture ofour preference trade-off management. The user requestsmedia from our central SIAM service and provides con-straints and rich preferences in the form of content pref-erences (UP) and technical capabilities (TC). The deci-sion engine negotiates the content by retrieving all nec-essary metadata integrating the respective transcodinghints (TH) into the preference trade-off management.

Fig. 5 Service-oriented Adaptation

The decision engine part of the SIAM service decidesby means of the given constraints which Web services arebasically needed to perform an adequate media delivery.Possible Web services (which are discovered by the ser-vice instantiation part) have to respect the preferences(and QoS constraints) to choose an optimal adaptationof the multimedia data for each individual user.The actual instantiation and monitoring engines used inour SIAM architecture are not the focus of this paperand have been extensively discussed in [1]. In a nutshell,the E2Mon (Execution and Environment Monitoring)framework allows a fast selection of new service chainsand a quality and cost based service failure recovery. Tofind an optimal service chain a cost function is used thatconsiders the parameters from the actual execution (suc-cess or failure) and furthermore a variety of costs de-pending on device capabilities or the execution environ-ment. If the costs of already discovered services change(e.g., the expected QoS), either a provider advertises thechange in a service or the environment has changed (e.g.,roaming to a new network or a low battery warning). Ineither case an event is raised and a rediscovery would

initiate the change to the cost-optimal solution, whilereusing the results of already invoked services when newbest service chains are derived.E2Mon is itself run as a Web service either on the clientor (more often) on a proxy machine to reduce the com-munication load of the client device for Web service in-vocation and monitoring. Of course, such a proxy-basedapproach is only useful if alarms (e.g. battery low) arerare compared to external events (service failures), asotherwise communicating the alarms can lead to a highcommunication load.

5 Efficiently Evaluating Preference Trade-Offs

To efficiently evaluate the complex mesh of the Paretoproduct order induced by preferences specially adaptedalgorithms are needed. Due to their practical usefulnessrecently in the field of databases some attention has beenpaid to concepts for retrieving Pareto optimal sets - so-called skyline queries, see e.g., [4] or [18].Skyline queries describe the case where all query at-tributes are considered to be independent and equallyimportant. Hence, no weighting function combining in-dividual attribute scores like in conventional in top-k re-trieval, can be used. Instead, all possibly optimal objects,based on the notion of Pareto optimality, are returned tothe user. An object is considered to be optimal with re-spect to a collection of objects and a set of preferences, ifit is not dominated by any other object. An object dom-inates another object, if it shows more or at least equallypreferred attribute values as this object with respect toall attributes and is strictly preferred in at least one at-tribute. However, skyline queries have been defined withtwo important constraints: the data has to be totallyordered for each attribute and for easy comparisons at-tributes have to be defined over numerical domains.Within skyline frameworks users are also offered the pos-sibility to declare several hard constraints on attributes,as given e.g., by transcoding hints or terminal capabili-ties. This is usually facilitated as a simple selection condi-tion for filtering [4]. But for the use in adaptation frame-works hard constraints have to be further distinguished.This is because some hard constraints can still be met byadapting content (like a codec or resolution constraint),whereas others can never be amended by altering a mediaobject (mostly usage preferences like preferred/dislikedactors or genres). We call hard constraints of the firsttype adaptation-sensitive hard constraints, whereas werefer to the second type as strict hard constraints.We realized the actual expression of explicit hard con-straints by preference tags containing an additional at-tribute ’constraint=”hard”’, which signalize the SIAMservice (and all subsequent Web services) that this pref-erence must not be violated.Since in our framework all preferences are based on par-tial orders, standard skyline query evaluation is thus not


readily applicable. Though it is possible to render a totalorder from partial order preferences by considering the’level’ of objects in each preference graph (the level be-ing defined by the longest path to any maximum valuein the graph), by doing so we accept some inaccuraciesinduced by object incomparability.

Example (cont.): For instance, taking the actorpreference of figure 3 we could transform the preferenceinto a total order with ’Damon’ as most important value(on level 1), followed by ’Willis’ and ’Schwarzenegger’on the same importance level 2, and finally have ’Pitt’and ’diCaprio’ as least preferred values again with sameimportance on level 3. But we would also state in thetotal order rendered from level information that ’Willis’is preferred over ’Pitt’ though in fact they are incompa-rable in the partial order preference.

But from this total order induced by levels it is sim-ple to derive a numerical assignment of score values forobjects by using a utility function, translating the levelinformation into simple scores. We normalize the scoreto a numerical value between 0 (least preferred) and 1(most preferred). For each respective attribute value ain the database a score SA(a) with respect to attributeA in the query expressed by total order preference p canbe computed, as follows:

SA(a) := (max level + 1− actual level)/(max level)

with max level as the total number of levels in preferencep and actual level as the level on which attribute valuea actually occurs with respect to A or (max level + 1),if the value a is not explicitly stated in p.

codec

actor

divx

mpeg

Huffyuv orIndeo

Pitt ordiCaprio

Willis or Schwarzenegger

Damon

optimal solution

Adaptation possible?

Fig. 6 Example Skyline for the Preferences of Figure 3

Example (cont.): Assume our user Kim requests amovie to be delivered to her PDA and states the prefer-ences in figure 3. Further assume that the database con-tains only the following objects: a ’Matt Damon’ moviein mpeg format (movie 1), an ’Arnold Schwarzenegger’movie in mpeg format (movie 2), a ’Bruce Willis’ moviein divx format (movie 3), a ’Leonardo diCaprio’ movie

in Huffyuv format (movie 4) and a ’Brad Pitt’ moviein mpeg format (movie 5). Figure 6 shows the resultingtuples. The skyline consists of only two objects (darkshaded): movie 1 and movie 3. An adaptation decisionengine never needs to consider any other object from thedatabase, i.e. movies 2, 4, and 5 (light shaded), becausethose are definitely dominated by the two skyline objects.For example, a ’Brad Pitt’ movie will never be needed(not even as a base for subsequent adaptation), becauseit will always be dominated by the existing ’Bruce Willis’movie in divx format with respect to both dimensions.

However, please note that according to the user pref-erences the most preferred object would be a ’Matt Da-mon’ movie in divx format (level 0 of the Pareto graph).But there simply is no instance in our database thatsatisfies this preference. In classical information systemsthe user query thus definitely has to be relaxed to theexisting skyline objects. On the other hand, in multi-media systems it is often possible to derive an optimalobject by suitable means. Though it is for instance notpossible in our example to create the most preferred ob-ject by adapting the Willis/divx, movie because we ob-viously cannot replace the actors, it is indeed possible totranscode the Damon/mpeg movie into a Damon/divxmovie (dotted arrow in figure 6), if there is a suitabletranscoding service available that meets all hard con-straints checked by the instantiation engine (e.g., QoSconstraints). Let us summarize some different possibili-ties:– Retrieve the Matt Damon movie in mpeg format and

transcode it into divx format before delivery.– Deliver the Bruce Willis movie in divx format.– Deliver the Matt Damon movie in mpeg format.– Retrieve the Arnold Schwarzenegger movie in mpeg

format and transcode it into divx format before de-livery.

– Deliver the Arnold Schwarzenegger movie in mpegformat

– ...Now we will take a closer look of how to compute a rank-ing for the best adaptation decision based on the moviesavailable in the database. Of course this ranking will alsohave to be according to the levels of the Pareto graph.Our scheme described in [3] presents a fair relaxationframework that can handle such problems. The frame-work describes how to relax soft constraints given by theuser if no exact match should be available. First the de-cision engine has to build a hierarchy (representing therelaxation path) out of all given preference graphs likeshown in figure 7.

The available tuples in our database are instances ofthe corresponding nodes (movie 1, movie 2, etc.). Thebest matching movie fulfills all soft constraints with aminimum amount of relaxation regarding a suitable qual-ity measure. Since the different nodes in the preferencegraphs are already ordered according to the necessary


PittdiCaprio

WillisSchwarzenegger

Damon

movie 4movie 5

movie 2movie 3

movie 1

HuffyuvIndeo

mpeg

divx

movie 4

movie 1movie 5movie 2

movie 3

1

2 2

1

Fig. 7 Preferences Relaxation Paths

relaxation steps from the top or the bottom of the hier-archy or simply the relative distance to the root nodes.We can use a simple weighting factor to all edges in ourhierarchy graphs to decide which movie to deliver. Ourbasic assumption for the computation of the weightingfactor is, the close we get to the root of a relaxation path(i.e. the more relaxation is needed), the more unsatisfiedthe user will be with the delivered result. We thus needto derive a quality measure to compare between mediaobjects.

Computation of the weighting factor (cf. [3]):Assume n preference graphs with their respective relax-ation hierarchies. Every edge is labeled by n(d−1), wheren is the number of preference graphs and d is the rela-tive distance to the root node to achieve a fair, breadthfirst relaxation scheme. Now each database object o islabeled with weighting factor wf(o) by summing up thelabels of edges relaxed to find this object in each hierar-chy. Sorting the object labels in ascending order leads toa preference order of media delivery decisions.

Example (cont.): In our example movie 1 and movie2 both got the same weighting of 1 (cf. figure 7). If theobjects have the same weightings and still no decision ispossible, we have a random choice between them. Thus,aiming at a minimum necessary amount of relaxation us-ing only the existing media according to we are left withtwo choices: delivering movie 1 (a Damon/mpeg movie)needs to relax the constraint on the codec by one leveland delivering movie 3 (a Willis/divx movie) needs torelax the constraint on the actor by one level.Before creating the final delivery order we have to takea look if it is possible to adapt one of the skyline moviesto become a better weighting factor, respectively a moreoptimal solution. The potential adaptability is a-prioryknown for each attribute. We know that it is impossi-ble to adapt the actor preference (figure 7, left) but it ispossible to change the codec preference (figure 7, right)by a suitable transcoding service. Please note that thedecision engine does not need to know, if such a serviceis actually available, because it only builds a hierarchy of

the best possible adaptation order. Coming back to ourexample, we see that it is not possible to improve theweighting factor for movie 3, but it is possible to adaptmovie 1 by transcoding. With a suitable transcoding ser-vice the weighting factor of movie 1* (Damon/divx) willbe 0 and becomes indeed an instance of the optimal so-lution of our preference problem (cf. figure 6).

Please note that as soon as there are media objectsalready suitable for immediate delivery in the list, nomore less important objects (i.e. not skyline objects) thathave to be created by adaptation are needed. Hence notall database objects have to be ranked, but really onlyskyline objects or objects created by suitable adapta-tion from skyline objects. The space of possible adapta-tions is therefore limited. The following algorithm givesan overview about the whole procedure:

1. /* Extract preferences according to their attributes */Given is the set P of preferences on n attributes inXML format by the user interaction tools.

1.1. Extract the hard constraints for each attributeHC1, ...,HCn

1.2. Divide HC1, ...,HCn in adaptation-sensitive hardconstraints HC1, ...,HCj and the strict hard con-straints HCj+1, ...,HCn

1.3. Extract the soft constraints for each attributeSC1, ..., SCn

2. /* Transform partial orders SCi(1 ≤ i ≤ n) into totalorders according to the respective level information.This step results in the preference relaxation graphs*/For each category i(1 ≤ i ≤ n) do

2.1. Compute max level for SCi as length of longestpath

2.2. For each attribute value a in SCi do2.2.1. Compute actual level for a as the maximum

path length to a from some top object in SCi

2.2.2. Compute SA(a) :=(max level + 1− actual level)/(max level)

2.3. Build total order SC ′i using the ranking informa-

tion given by SA(a)3. /* Determine possible media object set by skyline com-

putation and compute respective relaxation weights*/Get initial set O of available media objects from thedatabase using a skyline algorithm (e.g. from [4])with SC ′

i (1 ≤ i ≤ n) as input preferences and fil-tered by strict hard constraints HCj+1, ...,HCn

3.1. If O = ∅ then terminate with error/* No suitable (or at least adaptable) media objectis available in database */

3.2. For each object o ∈ O do3.2.1. Compute weighting factor wf(o) according to

preference relaxation graphs SC ′i (1 ≤ i ≤ n)

3.3. Set decision := false/* no adaptation decision taken yet */


4. /* Derive best adaptation decision */While (not decision) do

4.1. If O = ∅ then terminate with error/* No media object adaptation meets hard con-straints */

4.2. Choose set D ⊆ O of objects with currently lowestwf(o) (o ∈ D) as best available objects

4.3. If wf(o) = 0 (∀o ∈ D) then check adaptation-sensitive hard constraints HC1, ...,HCj

/* There is at least one perfect solution */4.3.1. If HC1, ...,HCj are satisfied for some o ∈ D

then set decision := true and return any suchobject o as adaptation decision/* successful termination */

4.3.2. Else discard set D from O and if O becomesempty terminate with error/* No media object is adaptable to meet hardconstraints */

4.4. /* try to adapt all objects with lowest wf(o) toobtain some object o’ with an even lower wf(o’)*/For each o ∈ D with wf(o) > 0 (o ∈ D) do

4.4.1. If o satisfies HC1, ...,HCj then put o into setD′ of possible adaptation decisions

4.4.2. Try all possible adaptations of o as given bySIAM workflows [1] and SC ′

1, , SC ′n to get ob-

ject o′ with wf(o′) < wf(o)4.4.2.1. If HC1, ...,HCj are satisfied for o′ then

put o′ into set D′ of possible adaptationdecisions, else discard o′

4.4.3. Discard o from D and O4.5. If D′ 6= ∅ get object x from D′ with minimum

wf(x), set decision := true and return x as bestmatching adaptation decision/* successful termination */

The decision engine can now derive the final adapta-tion order by sorting the skyline objects and their pos-sible adaptations as output by the algorithm accordingto their respective weighting factor. It then hands therespective list over to the service instantiation engine.This engine takes the sole responsibility for actually se-lecting the service chain by iterating over the adaptationlist and checking individually whether a suitable chaincan be instantiated at the current time.However, one problem that may occur when using sky-line query processing is that Pareto sets are known togrow exponentially in size with increasing number of in-dependent preferences. As a remedy in [2] we introducedan approach to significantly reduce the skyline size by us-ing so-called prime cuts. Prime cuts are interesting sub-sets of the full Pareto skyline, which are representative ofthe respective skyline and generally provide good com-promises. The key to improved performance and reducedresult sets sizes here is the relaxation of Pareto seman-tics to the concept of weak Pareto dominance (see [2] fordetails). The set of all non-weakly-dominated objects isreferred to as the ’restricted’ skyline.

In comparison to the full Pareto skylines our evaluationover a database with 25000 items in [2] shows, that by us-ing restricted skylines the number of tuples only slightlyincreases, if more preferences are stated (see figure 8).Moreover, even for large numbers of preferences the re-stricted skyline can be computed about two orders ofmagnitude faster than traditional Pareto skylines (seefigure 9). We can state that the restricted skyline ap-proach makes even high-dimensional preference trade-offs (as may sometimes be necessary in personalized me-dia adaptation) practical.

Fig. 8 Preference dimensionality and skyline size

Fig. 9 Preference dimensionality and runtimes

6 A Media Streaming Test Case

As a proof of concept we implemented the key featuresof the decision engine in a small testbed (called PUMA)based on Web services. Since we were interested in de-riving suitable adaptations our implementation focusedspecifically on the changes in the metadata documentsand what could be derived as final adaptation decisions.We did not perform the actual adaptation of the videofiles in the testbed. All Web services have been developed


in Microsoft Visual Studio .NET in C#. For user inter-action the SIAM (service instantiation and monitoring)service is addressed by a simple PHP interface, whereusers can place their initial requests with adequate pref-erences. Like discussed in section 4, the SIAM serviceruns the E2Mon algorithm from [1].Figure 10 shows the elements of our basic testbed. Theexperimental scenario focused different typical scenarioslike the retrieval of videos from alternative databases,the merging with subtitles in different languages, as wellas transcoding and quality adaptation. The initial queryparameters, the user preferences and terminal capabili-ties are written into a MPEG-21 document, which is sentto the central SIAM service for further processing.

Subtitle DB

Video MetadataDB1

Video MetadataDB2

SubtitleService

WebService

DB1

WebService

DB2

SIAMService

PHPClient

QualityAdaptation

ServiceMergingService

Transco-ding

Service

Fig. 10 Experimental Scenario

Figure 11 shows a screenshot of the PUMA testbed.Attribute values for (hard) content preferences or prefer-ence information in terms of profiles (like the PDA tech-nical profile) can be submitted to the system. Objectsare selected from the database accordingly and then pre-pared for further processing with an appropriate meta-data description file stating all necessary details.

Example (cont.): Let us come back to user Kimwho wants a movie streamed to her PDA with her pref-erences as discussed before and additionally requests acertain scaling according to her client device’s displaysize specified in the PDA profile.

As discussed in section 5 the service instantiation en-gine gets the list from the adaptation decision engineand will choose an adequate workflow from a set of pat-terns. It first tries the transcoding of the ’Matt Damon’movie into divx format to deliver the best possible solu-tion according to the preferences stated by Kim. To dothis it has to integrate a suitable transcoding service intothe delivery workflow. Since in most multimedia appli-cations workflows can be anticipated and are well under-stood this task is performed by checking if an adequatetranscoding service is available by requesting the func-tionality using a UDDI service directory. If the enginefinds a suitable service, it will try to instantiate the cor-

Fig. 11 Screenshot of the PUMA testbed

responding service chain.By handing over the preference constraints (includingespecially all hard constraints) the transcoding serviceis now enabled to check, if it is possible to transcodethe movie into divx, e.g. matching statistics on the ex-pected time needed for transcoding, etc. against the user-provided constraints. The service monitoring engine mon-itors the whole adaptation process. Hence, if the transcod-ing service decides that a transcoding is not possibleor the transcoding fails, e.g. because the service is notlonger available, a respective alarm is raised to the moni-toring engine as described in detail in [1]. Now the serviceinstantiation engine will try to find an equivalent serviceto recover from failure or decide that an adaptation ofthe Damon/mpeg movie to a Damon/divx movie is gen-erally not possible. In this case it will try to instantiatethe next alternative from the adaptation list.If the transcoding succeeds, a scaling service (figure 9:quality adaptation service) is still needed, to adapt thecurrent resolution of the movie to the highest possibledisplay resolution of the PDA (see figure 2). This serviceagain gets all preference data and decides locally if it issensible to scale down the resolution. To make a decisionabout that, the Web service for example compares theinformation from the transcoding hints to the terminalcapabilities. If a mismatch occurs, again an alarm hasto be raised. Please note, that though some checks canalready be performed by the adaptation decision engine,the services always may have to integrate local param-eters into the decision whether to perform a task andwhat is the best way.Moreover, in a typical workflow Kim may also requestsubtitles for a certain movie. The subtitles then can alsobe saved into the MPEG-21 metadata document andhanded on to a merging service together with the origi-nal media from the database. After the merging processthe service removes the now obsolete subtitles and the


language preferences from the metadata document of theadapted movie. and finally the adapted movie is deliv-ered to user Kim.Throughout our experiments it proved possible to han-dle the preference matchmaking and thus create a dy-namic service composition based on the metadata withthe SIAM service as central coordinator.

7 Conclusions and Future Work

With the increasing need of service oriented conceptsin complex multimedia applications also the adaptationof content is split into several tasks. Depending on theavailable media, the discovered services and all seman-tic, user-specific and technical constraints, the selectionof the best possible workflow needs to solve a complexpreference problem. The current preference model in theMPEG-7/21 standard, however, is limited to the sim-ple matching of numerical values representing the im-portance of each constraint. Hence for complex instanti-ations of workflows in a service-oriented adaptation en-gine a more sophisticated preference model is needed.By integrating a model of partial order preferences andevaluating complex preference constraints using the sky-line queries as currently explored by the database com-munity, our framework is able to handle trade-offs ina more meaningful way. We have shown on the exam-ple of a typical media streaming scenario that an au-tomatic adaptation of the multimedia content is possi-ble by means of structured metadata information andhow to efficiently explore the complex result meshes thatemerge from combining independent content and adap-tation preferences. These preferences are collected fromindependent sources such as explicit user preferences, in-trinsic domain preferences, transcoding hints, or techni-cal device capabilities. This complete preference infor-mation can then be augmented by all Web services inthe workflow and thus allows an intelligent and highlypersonalized adaptation process. Moreover, every singleWeb service is enabled to decide how to adapt multime-dia material best.After implementing the key components as a proof ofconcept we are currently implementing the entire frame-work including the actual adaptation of video streamsto get further insights on the scalability of our approach.The MPEG-21 standard already provides some descrip-tion schemes to contain network QoS parameters, whichwill have a strong influence on the scalability of service-oriented adaptation processes. We want to explore inhow far these parameters can be measured or estimatedaccurately enough to guarantee stable service composi-tions (especially since some parameters like e.g. networklatency, may change very quickly and lead to the consid-eration of alternative service chains).Another open question is, if the routing path should alsobe determined by preferences. Generally speaking, rout-

ing media can be performed in a custom-made overlayallowing the distribution of services along the routingpath. Here, the effective management of trade-offs forall services involved can lead to a better overall ser-vice for end users. Finally, we plan to integrate higherlevel semantic descriptions into our framework. Recentlythe World Wide Web Consortium (W3C) started a Se-mantic Web task force [26] to investigate multimedia an-notations. The respective standardization activities arepromising to overcome the problem of missing mappingfunctions for MPEG-7/21.

References

1. W.-T. Balke, J. Diederich: A Quality- and Cost-based Se-lection Model for Multimedia Service Composition in Mo-bile Environments. In IEEE Int. Conf. on Web Services(ICWS), Chicago, IL, USA, 2006.

2. W.-T. Balke, U. Guntzer, W. Siberski: Getting PrimeCuts from Skylines over Partially Ordered Domains. InDatabase systems in Business, Technologie and Web(BTW 2007), Aachen, Germany, 2007.

3. W.-T. Balke, M.Wagner: Through Different Eyes - Assess-ing Multiple Conceptual Views for Querying Web Services.In World Wide Web Conference (WWW), New York,USA, 2004.

4. S. Borzsonyi, D. Kossmann, K. Stocker: The Skyline Op-erator. In International Conference on Data Engineering(ICDE), Heidelberg, Germany, 2001.

5. P Brusilovsky, A Kobsa, J Vassileva: Adaptive Hyper-Text and Hypermedia. Kluwer Academic Publishers, USA,1998.

6. J. Chomicki: Preference Formulas in Relational Queries.In ACM Transactions on Database Systems (TODS), Vol.28 (4), 2003.

7. X. Gu, K. Nahrstedt, R. Chang, C. Ward: QoS-assuredService Composition in Managed Service Overlay Net-works, In IEEE ICDCS 2003, Providence, RI, USA, 2003.

8. W. Kellerer, M. Wagner, W.-T. Balke, H. Schulzrinne:Preference-based Session Management for IP-based Mo-bile Multimedia Signaling. In European Transac-tions onTelecommunications, Vol. 15 (4), Wiley, 2004.

9. W. Kießling: Foundations of Preferences in Database Sys-tems. In Int. Conf. on Very Large Databases (VLDB),Hong Kong, China, 2002.

10. A. Kobsa, W. Wahlster: User models in dialog systems.Springer, USA, 1990.

11. B. Kohncke, W.-T. Balke: Personalized Digital ItemAdaptation in Service-Oriented Environments. In Int.Workshop on Semantic Media Adaptation and Personal-ization (SMAP), Athens, Greece, 2006.

12. H.-K. Lee, J. Nam, B. Bae, M. Kim, K. Kang, J. Kim:Personalized Contents Guide and Browsing based on UserPreference. In AH2002 Workshop on Personalization inFuture TV, Malaga, Spain, 2002.

13. C.-S. Li, R. Mohan, J. R. Smith: Multimedia ContentDescription in the InfoPyramid. In IEEE Int. Conf. onAcoustics, Speech and Signal Processing, Seattle, WA,USA, 1998.

14. D. Mukherjee, E. Delfosse, J.-G. Kim, Y. Wong: Opti-mal Adaptation Decision-Taking for Terminal and Net-work Quality-of-Service. In IEEE Transactions on Multi-media, Vol. 7 (3), 2005.

15. P. Mylonas, M. Wallace: Using Ontologies and FuzzyRelations in Multimedia Personalization. In Int. Work-shop on Semantic Media Adaptation and Personalization(SMAP), Athens, Greece, 2006.


16. K. Nahrstedt, D. Xu: Finding Service Paths in a MediaService Proxy Network. In SPIE/ACM Multimedia Com-puting and Networking Conference (MMCN), San Jose,CA, USA, 2002.

17. K. Nahrstedt, W.-T. Balke: Towards Building LargeScale Multimedia Systems and Applications: Chal-lengesand Status. In ACM Workshop on Multimedia ServiceComposition (MSC), Singapore, 2005.

18. D. Papadias, Y. Tao, G. Fu, B. Seeger: An optimal andprogressive algorithm for skyline queries. In ACM SIG-MOD Conference (SIGMOD 03), San Diego, USA, 2003.

19. G. Stamou, J. van Ossenbruggen, J. Z. Pan, G. Schreiber:Multimedia Annotations on the Semantic Web. In IEEEMultimedia, Vol. 13(1), 2006.

20. B. L. Tseng, C.-Y. Lin, J. R. Smith: Using MPEG-7 andMPEG-21 for Personalizing Video. In IEEE Multimedia,Vol. 11(1), 2004.

21. C. Tsinaraki, S. Christodoulakis: Semantic User Prefe-rence Descriptions in MPEG-7/21. In Hellenistic DataManagement Symposium (HDMS), Athens, Greece, 2005.

22. C. Tsinaraki, S. Christodoulakis: A User PreferenceModel and an Query Language that Allow Semantic Re-trieval and Filtering of Multimedia Content. In Int. Work-shop on Semantic Media Adaptation and Personalization(SMAP), Athens, Greece, 2006.

23. P. van Beek, A. B. Benitez, J. Heuer, J. Martinez, P.Salembier, J. Smith, T. Walker: MPEG-7 Multimedia De-scription Schemes (Version 3.1, Working Draft), ISO/IEC,2000.

24. A. Vetro: MPEG-21 Digital Item Adaptation: EnablingUniversal Multimedia Access. In IEEE Multimedia, Vol.11 (1), 2004.

25. A. Vetro, C. Timmerer: Digital Item Adaptation: Over-view of Standardization and Research Activities. In IEEETransactions on Multimedia, Vol. 7 (3), 2005.

26. Word Wide Web Consortium (W3C):Multimedia An-notation on the SemanticWeb.http://www.w3.org/2001/sw/BestPractices/MM/

27. M. Zhou, V. Aggarwal: An Optimization-based Approachto Dynamic Data Content Selection in Intelligent Multi-media Interfaces. In ACM Symposium on User InterfaceSoftware and Technology, Santa Fe, NM, USA, 2004.

Benjamin Kohncke currently is aMaster student at University of Han-nover and has received his BA inComputer Science at the Universityof Hannover, Germany in 2006.

Wolf-Tilo Balke since 2004 asso-ciate research director of L3S Re-search Center of University of Han-nover. Before that he was a researchfellow at the University of Californiaat Berkeley, USA. He is a memberof the Emmy-Noether-Program of theGerman Research Foundation. He hasreceived his MS in mathematics anda PhD in computer science from Uni-versity of Augsburg, Germany.

Documents

Wolf-Tilo Balke Preference-Driven Personalization for ... · on what MPEG-7/21 can actually express in user pref-erences. The User Interaction Tools (UIT) oﬀer several possibil-ities