Enrichment of News Show Videoswith Multimodal Semi-Automatic AnalysisDaniel Stein1, Evlampios Apostolidis2, Vasileios Mezaris2, Nicolas de Abreu Pereira3,Jennifer Müller3, Mathilde Sahuguet4, Benoit Huet4, and Ivo Lašek5

Fraunhofer Institute IAIS, Sankt Augustin, Germany1 Informatics and Telematics Institute CERTH, Thermi-Thessaloniki, Greece2 Rundfunk Berlin-Brandenburg, Potsdam, Germany3 Eurecom, Sophia Antipolis, France4

Czech Technical University in Prague, Prague, Czech Republic5

Abstract: Enriching linear videos by offering continu-ative and related information via, e.g., audiostreams,webpages, as well as other videos, is typically hamperedby its demand for massive editorial work. While thereexist several automatic and semi-automatic methods thatanalyse audio/video content, one needs to decide whichmethod offers appropriate information for an intendeduse-case scenario. In this paper, we present the newsshow scenario as defined within the LinkedTV project,and derive its necessities based on expected user arche-types. We then proceed to review the technology op-tions for video analysis that we have access to, and de-scribe which training material we opted for to feed ouralgorithms. Finally, we offer preliminary quality feed-back results and give an outlook on the next steps withinthe project.

Keywords: Multimodal Video Analysis, News showscenario, LinkedTV

1 IntroductionMany recent surveys show an ever growing increase in av-erage video consumption, but also a general trend to si-multaneous usage of internet and TV: for example, cross-media usage of at least once per month has risen to morethan 59% among Americans [5]. A newer study [13] evenreports that 86% of mobile internet users utilize their mo-bile device while watching TV. This trend results in con-siderable interest in interactive and enriched video experi-ence, which is typically hampered by its demand for mas-sive editorial work.

A huge variety of techniques, both new and established,exist that analyse video content (semi-)automatically. Ide-ally, the processed material then offers a rich and pervasivesource of information to be used for automatic and semi-automatic interlinking purposes. However, the informationproduced by video analysis techniques is as heterogeneousas is their individual approach and the expected complex-ity, which is why careful planning is crucial, based on thedemands of an actual use-case scenario. This paper intro-duces a news broadcast scenario for video enrichment asenvisioned in the EU-funded project “Television linked tothe Web” (LinkedTV),1 and analyses the needs for practi-cal usage. These needs form the basis for the technologythat we employ for video analysis, and for the databasesthat we use to feed the training algorithms. We presentour decisions on automatic speech recognition, keywordextraction, shot/scene segmentation, concept detection, thedetection and tracking of moving and static objects, as wellas unsupervised face clustering. Finally, we offer prelimi-nary results based on human feedback.

1www.linkedtv.eu

This paper is structured as follows: we present the en-visioned news show scenario of LinkedTV (Section 2), de-scribe the analysis techniques and training material that arecurrently used (Section 3), provide manual examination offirst experimental results (Section 4), and finally elaborateon future directions that will be pursued (Section 5).

1.1 Related WorkWe will continue to review two recent projects with a re-lated overall focus as LinkedTV. For the more detailed an-alytic methods as listed in Section 3, we will give appropri-ate citations along with their descriptions later in the paper.

HBB-NEXT2 is a EU-funded project which aims atlinking internet and broadcast, but its main focus are tech-nological aspects of a hybrid multimedia service, i.e., device-independent applications, synchronisation for service com-ponents and multi-device viewing, for example via sec-ond screen applications. While these aspects are partly in-cluded in LinkedTV as well, HBB-NEXT does not workon (semi-)automatic enrichment of a given video collec-tion.

Projects with similar focus include, e.g., inEvent3, whichis another EU-funded project on video material analysisand searchability, using A/V processing techniques enrichedwith semantics, and recommendations based on social net-work information. However, the projects’ main targets aremeetings, video-conferences and lectures, which are morerestricted in a sense that they only include limited personelland domains within one video.

2 News Broadcast ScenarioThe audio quality and the visual presentation within videosfound in the web, as well as their domains, are very hetero-geneous. To cover many aspects of automatic video anal-ysis, we have identified several possible scenarios for in-terlinkable videos within the LinkedTV project, e.g., (a) anews show, (b) cultural heritage, and (c) visual arts [11]. Inthis paper, we focus on the specific demands of the newsshow scenario. The scenario uses German news broadcastas seed videos, provided by Public Service BroadcasterRundfunk Berlin- Brandenburg (RBB).4 The main newsshow is broadcast several times each day, with a focus onlocal news for Berlin and Brandenburg area. For legal andquality reasons, the scenario is subject to many restrictionsas it only allows for editorially controlled, high qualitylinking. For the same quality reason only links selectedfrom a restricted whitelist are allowed. This whitelist con-tains, for example, videos produced by the Consortium of

2www.hbb-next.eu3www.inevent-project.eu4www.rbb-online.de

public service broadcasting institutions of the Federal Re-public of Germany (ARD) and a limited number of ap-proved third party providers.

The audio quality of the seed content can generally beconsidered to be clean, with little use of jingles or back-ground music. Interviews of the local population may havea minor to thick accent, while the eight different modera-tors have a very clear and trained pronunciation. The mainchallenge for visual analysis is the multitude of possibletopics in news shows. Technically, the individual elementswill be rather clear: contextual segments (shots or stories)are usually separated by visual inserts and the appearanceof the anchorman/-woman, and there are only few quickcamera movements.

2.1 Scenario ArchetypesLinkedTV envisions a service that offers enriched videoswhich are interlinked with the web, and targets a broaderaudience. For the sake of a convincing scenario, however,we have sketched three archetypal users of the LinkedTVnews service and their motivations to use it:

Ralph comes home from working on a building site inPotsdam, and starts watching “rbb AKTUELL”. The firstspots are mainly about politics and about Berlin. Ralph isnot particularly interested, neither in politics nor in Berlinas he lives in a small town in Brandenburg. After a whilethere is the first really interesting news for Ralph: a spotabout the restoration of a church at a nearby lake; as acarpenter, Ralph is always interested in the restoration ofold buildings. Therefore, he watches the main news spotcarefully and views an extra video and several still imagesabout the church before and after its restoration. Finally,the service also offers links to a map and the website of thechurch which was set up to document the restoration fordonators and anyone else who would be interested. Ralphsaves these links to his smartphone so he can visit the placeon the weekend.

Nina’s baby has fallen asleep after feeding, so she findssome time for casually watching TV, to be informed whiledoing some housework. Browsing the programme she seesthat yesterday’s enhanced “rbb AKTUELL” evening edi-tion is available and starts the programme. Nina watchesthe intro with the headlines while starting her houseworksession with ironing some shirts. Watching a news spotabout Berlin’s Green Party leader who withdrew from hisoffice yesterday, Nina is kind of frustrated as she voted forhim and feels her vote is now “used” by someone she mightnot have voted for. She would like to hear what other politi-cians and people who voted for him think about his deci-sion to resign. She watches a selection of video statementsof politicians and voters and bookmarks a link to an on-line dossier about the man and his political carrier whichshe can browse later on her tablet. Eventually, the babyawakes so Nina pauses the application so she can continuelater.

Socially active retiree Peter watches the same newsshow with different personal interest and preferences. Oneof the spots is about a fire at famous Café Keese in Berlin.Peter is shocked. He used to go there every once in a while,but that was years ago. As he hasn’t been there for ages, hewonders how the place may have changed over the years.In the news spot, smoke and fire engines was almost all onecould see, so he watches some older videos about the storyof the famous location where men would call women ontheir table phones – hard to believe nowadays, he thinks,

Figure 1: Screenshot of the EXMARaLDA GUI, withautomatically derived information extracted from an rbbvideo.

now that everyone carries around mobile phones! Afterchecking the clips on the LinkedTV service, he returns tothe main news show and watches the next spot on a newInternet portal about rehabilitation centres in Berlin andBrandenburg. He knows an increasing number of peoplewho needed such facilities. He follows a link to a mapof Brandenburg showing the locations of these centres andbookmarks the linked portal website to check some moreinformation later. At the end of the show, he takes an in-terested look at the weather forecast, hoping that tomorrowwould be as nice as today so he could go out again to baskin the sun.

3 Technical BackgroundNow that we have defined the motivation and needs of thearchetypes above, we need access to a very heterogeneousset of information to be (semi-)automatically derived fromthe A/V content. We will proceed to list the technologyemployed to address these requirements. This section isdivided into four sub-parts, being automatic speech recog-nition, temporal segmentation, spatiotemporal segmenta-tion, person recognition, and other meta-information.

All this material is joined in a single xml file for eachvideo, and can be visualized and edited by the annotationtool EXMARaLDA [8]. Note that for the time being, EX-MARaLDA does not support spatial annotation; we plan toextend the tool at a later stage of the project.

3.1 Automatic Speech RecognitionWhenever there are no subtitles available, an automaticspeech recognizer is needed in order to employ keywordextraction as well as named entity recognition. If subti-tles are given, forced alignment techniques to match thetimestamps to the video on a word level could be useful,because the utterance timestamp provided by the subtitlesmight be to imprecise and coarse-granular for our needs.In both cases, we need a strong German acoustic model.We employ a state-of-the-art speech recognition system asdescribed in [9]. For training of the acoustic model, weemploy 82,799 sentences from transcribed video files. Inaccordance with the news show scenario, they are takenfrom the domain of both broadcast news and political talkshows. The audio is sampled at 16 kHz and can be consid-ered to be of clean quality. Parts of the talk shows are omit-ted when, e.g., many speakers talk simultaneously or whenmusic is played in the background. The language modelconsists of the transcriptions of these audio files, plus ad-

ditional in-domain data taken from online newspapers andRSS feeds. In total, the material consists of 11,670,856sentences and 187,042,225 running words. Of these, theindividual subtopics were used to train trigrams with modi-fied Kneser-Ney discounting, and then interpolated and op-timized for perplexity on a with-held 1% proportion of thecorpus.

3.2 Temporal SegmentationLarger videos should be temporally segmented based ontheir content. In our scenario, this would enable to skipparts of the video and directly jump to the weather fore-cast, for example. Also, we need to provide reasonabletimestamp limits for hyperlinks, since we do not want fur-ther information on Berlin’s Green Party to still be activeonce the clip about the rehabilitation center starts.

Currently, we segment the video into shots (fine-granulartemporal segments which relate to, e.g., the camera an-gle) and scenes (coarse-granular temporal segments whichdraw input from shot segmentation and performs shot group-ing into sets).

Video shot segmentation is based on an approach pro-posed in [12]. The employed technique can detect bothabrupt and gradual transitions; however, in certain use cases(depending on the content) it may be advantageous for min-imizing both computational complexity and the rate of falsepositives to consider only the detected abrupt transitions.Specifically, this technique exploits image features such ascolor coherence, Macbeth color histogram and luminancecenter of gravity, in order to form an appropriate featurevector for each frame. Then, given a pair of selected suc-cessive or non-successive frames, the distances betweentheir feature vectors are computed, forming distance vec-tors, which are then evaluated with the help of one or moreSVM classifiers. In order to further improve the results, weaugmented the above technique with a baseline approach toflash detection. Using the latter we minimize the numberof incorrectly detected shot boundaries due to camera flasheffects.

Video scene segmentation groups shot segments intosets which correspond to individual scenes of the video.The employed method was proposed in [10]. It introducestwo extensions of the Scene Transition Graph (STG) al-gorithm; the first one aims at reducing the computationalcost of shot grouping by considering shot linking transitiv-ity and the fact that scenes are by definition convex setsof shots, while the second one builds on the former toconstruct a probabilistic framework towards multiple STGcombination. The latter allows for combining STGs builtby examining different forms of information extracted fromthe video (i.e., low level audio or visual features, visualconcepts, audio events) while at the same time alleviatingthe need for manual STG parameter selection.

3.3 Spatiotemporal SegmentationObjects of interest should be detected and tracked so thatthey can be clicked on for further information. Due to thebroad target domains it cannot be guaranteed that estab-lished databases contain enough instances for local enti-ties, which is why we need strong clustering and re-detectiontechniques so that an editor only needs to label them onceand can automatically find other instances within the videoitself, or within a larger set of surrounding videos.

For the purpose of spatiotemporal segmentation of the

video stream, we differentiate between moving and staticobject detection. Spatiotemporal segmentation of a videoshot into differently moving objects is performed as in [2].This unsupervised method uses motion and color infor-mation directly extracted from the MPEG-2 compressedstream. The bilinear motion model is used to model themotion of the camera (equivalently, the perceived motionof static background) and, wherever necessary, the motionof the identified moving objects. Then, an iterative rejec-tion scheme and temporal consistency constraints are em-ployed for detecting differently moving objects, account-ing for the fact that motion vectors extracted from the com-pressed stream may not accurately represent the true objectmotion.

For detecting occurrences of static objects of interestin consecutive or non consecutive video frames, like thechurch or the café in the scenario, a semi-automatic ap-proach based on object re-detection will be adopted. Theuser will manually specify the object of interest by markinga bounding box on one frame of the video. Then, the ad-ditional instances of the same object in subsequent frameswill be automatically detected via object matching. Match-ing between image regions will be performed based onSURF descriptors [1] and some geometric restrictions de-fined by the RANSAC algorithm [2]. A baseline OpenCVimplementation will initially be adopted for this. For eachpair of images, feature vectors will be extracted using theSURF algorithm and will be compared. False matches willbe filtered out using a symmetrical matching scheme be-tween the pair of images, and the remaining outliers willbe discarder by applying the epipolar constraint calculatedfrom the RANSAC method.

3.4 Person DetectionPersons seen or heard within a video are arguably the mostimportant information for a viewer. They should be identi-fied via their face and/or their voice, i.e., one can rely bothon face detection as well as speaker recognition. For localcontent where some of the persons might be unknown withregard to the training material, there is also demand for un-supervised person clustering similar to objects of interestabove, so that, e.g., the former leader of Berlin’s GreenParty needs only to be labelled once by a human editor.

3.4.1 Face Analysis

Faces analysis is performed on keyframes extracted fromthe video (temporal subsampling), and employs the face.comAPI.5 The process can be divided into three components:face detection, face clustering and face recognition. Facedetection is used as a prior tool to perform the other tasks,which both stem from the calculation of the similarity be-tween detected faces. Our implementation is based on aniterative training/recognition approach [3] to make accu-rate groupings: the training process is initialized with thefirst detected face in the video. For each subsequent pic-ture, the detected faces are matched against the initial face.If the recognition confidence level is higher than a thresh-old (80% performed well in our experiments), both facesare associated with the same id, otherwise a new face idis created. After every assignment, the corresponding facemodel is retrained before performing recognition on thenext candidate face. We output our results in a xml filethat provides the coordinate of the faces detected in each

5http://developers.face.com

picture (center, length and width of the bounded-box), to-gether with the id of the face (id of the cluster). In orderto guaranty the highest accuracy possible, we rely on a hu-man annotator to label the face clusters with the appropri-ate person name.

3.4.2 Speaker Recognition

For speaker identification (SID), we follow the approach of[6], i.e., we make use of Gaussian Mixture Models (GMMs)using spectral energies over mel-filters, cepstral coefficientsand delta cepstra of range 2. An overall universal back-ground model (UBM) is merged from gender-dependentUBMs and forms the basis for the adaptation of person-dependent SID models. To train and assess the quality ofthe speaker identification, we listed German politicians asa possible requirement within our scenario description inSection 2.1. Thus, we downloaded a collection of speechesfrom 253 German politicians, taken from the archive of theGerman parliament.6 In total, this consists of 2581 fileswith 324 hours of training material. To make training ofthe models feasible, we use 2 minutes per file to adapt theUBM.

3.5 Meta-InformationFor keywords needed to tag the videos, they can eitherbe extracted from textual sources, or derived from video-based concept detections. These tags can then be used torecommend similar videos like, e.g., of churches in the lo-cal area.

3.5.1 Concept Detection

A baseline concept detection approach is adopted from [4].Initially, 64-dimension SURF descriptors are extracted fromvideo keyframes by performing dense sampling. These de-scriptors are then used by a Random Forest implementa-tion in order to construct a Bag-of-Words representation(including 1024 elements) for each one of the extractedkeyframes. Following the representation of keyframes byhistograms of words, a set of linear SVMs is used for train-ing and classification purposes, and the responses of theSVM classifiers for different keyframes of the same shotare appropriately combined. The final output of the classi-fication for a shot is a value in the range [0, 1], which de-notes the Degree of Confidence (DoC) with which the shotis related to the corresponding concept. Based on theseclassifier responses, a shot is described by a 232-elementmodel vector, whose elements correspond to the detectionresults for the 323 concepts defined in the TRECVID 2011SIN task.7 These 323 concepts resulted from the originally346 concepts defined in TRECVID 2011, after discardinga few that are either too generic (e.g., “Eukaryotic Organ-ism”) or irrelevant to the current data being considered inLinkedTV (cf. [11]).

Moreover, in order to improve the detection accuracywe used some relations between concepts as they were de-termined in TRECVID 2011. When a concept implies an-other concept (e.g., the concept “Man” implies the concept“Person”) then the confidence level of the second conceptis reinforced by adding the joined confidence score by anempirically set factor α . On the contrary, when a concept

6http://webtv.bundestag.de/iptv/player/macros/bttv/index.html

7www-nlpir.nist.gov/projects/tv2011/

1 2 5 10 20 40 1

2

5

10

20

40

False Alarm probability (in %)

Mis

s p

rob

ab

ility

(in

%)

Speaker Detection Performance

128 GMMs

256 GMMs

512 GMMs

1024 GMMs

Figure 2: Speaker identifaction for German politians:DET Curve for different mixture sizes of the GMM, ona withheld test corpus of 994 audio files from the Germanparliament.

excludes another concept (e.g., the concept “Daytime Out-door” excludes the concept “Nightime”) and if the confi-dence score of the first concept is higher than the second,the first one is enhanced and the second one is penalizedaccordingly.

3.5.2 Keyword Extraction

The objective of keyword extraction or glossary extractionis to identify and organize words and phrases from doc-uments into sets of glossary-items or keywords. For thisparticular scenario, we have access to several sources oftextual information about a particular video. These in-clude subtitles, manual annotations of videos, and finallythe transcripts obtained from the ASR. Since LinkedTV isa multilingual project, we decided to refrain from using lin-guistically based, i.e., language dependent techniques herebut employ a statistical approach based on text frequency –inverse document frequency (TF-IDF) [7] weights of wordsextracted from videos.

4 ExperimentsIn this section, we present the result of the manual evalua-tion of a first analysis of “rbb AKTUELL” videos.

The ASR system produces reasonable results for thenews anchorman and for reports with predefined text. Ininterview situations, the performance drops significantly.Further problems include named entities of local interest,and heavy distortion when locals speak with a thick dialect.We manually analysed 4 sessions of 10:24 minutes total(1162 words). The percentage of erroneous words wereat 9% and 11% for the anchorman and voice-over parts,respectively. In the interview phase, the error score roseto 33%, and even worse to 66% for persons with a localdialect.

To evaluate the quality of the SID, a distinct set of 994audio files has been used to evaluate the quality of the mod-els. A GMM with 128 mixtures has an Equal Error Rate(EER) of 9.86, whereas using 1024 mixtures leads to animprovement of 8.06 EER. See Figure 2 for Detection Er-ror Trade-Off (DET) curves.

Figure 3: Clusters 1, 6, 16 and 18 of the face clusteringresult. Clusters 1 and 18 contain the anchorman and nonoise face: two groups are made depending on the angle ofhis head. We have the same effect for clusters 6 and 16.

We performed face clustering on 200 shots extractedfrom the seed video at regular intervals. Results give a to-tal of 54 clusters, most of which containing a single face(people that appear once). All 54 clusters are pure (i.e.,contain only 1 person’s face), and 2 ids (persons) appearin more than 1 cluster (as seen in Figure 3) due to signifi-cant viewing angle difference. Such clusters can easily beprocessed by an annotator. To make this process easier, weconsider to let aside the clusters that contain only one faceimage, on the assumption that a person that appears onlyonce is not of primary importance for the video.

In preliminary experiments on shot segmentation, zoom-ing and shaky cameras were misinterpreted as the begin-nings of new shots. Also, reporters’ flashlights confusedthe shot detection. In total, the algorithm detected 306shots, which are mostly accurate on the video level butwere found to be too fine-granular for our purposes. Ahuman annotator only marked one third, i.e., 103 temporalsegments, to be of immediate importance. In a second iter-ation with conservative segmentation, most of these issuescould be addressed.

Indicative results of spatiotemporal segmentation onthese videos, following their temporal decomposition toshots, are shown in Figure 4. In this figure, the red rect-angles demarcate automatically detected moving objects,which are typically central to the meaning of the corre-sponding video shot and could potentially be used for link-ing to other video, or multimedia in general, resources.Currently, an unwanted effect of the automatic process-ing is the false recognition of name banners which slidein quite frequently during interviews, which indeed is amoving object but does not yield additional information.

Manually evaluating the top-10 most relevant conceptsaccording to the classifiers’ degrees of confidence revealedthat the concept detectors often succeed in providing usefulresults; yet there is significant room for improvement. SeeFigure 5 for two examples, the left one with good results,

Figure 4: Spatiotemporal Segmentation on video samplesfrom news show “RBB Aktuell”

and the right one with more problematic main concepts de-tected.

5 ConclusionIn this paper, we presented the news show scenario as pur-sued by the LinkedTV consortium. We have access tostate-of-the-art techniques that can analyse the video con-tent in order to derive the needed information, and reportedreasonable preliminary results.

The main challenge will be to interweave the singleresults into refined high-level information. For example,the person detection can gain information from automaticspeech recognition, speaker recognition and face recogni-tion. Also, in order to find reasonable story segments ina larger video, one can draw knowledge both from speechsegments, topic classification, and video shot segments. Asa final example, video similarity can be estimated with fea-ture vectors carrying information from the concept detec-tion, the keywords, the topic classification and the entitiesdetected within the video.

We already collected all (semi-)automatically producedinformation bits into a single annotation file that can beviewed with a proper tool, and will use this in order toestablish ground full truth material on the scenario datain a next step. Then, we plan to validate and extend themethods to increase their accuracy. Finally, we hope togain more insight from the multimodal feature combina-tion, which includes combining different confidence scoresfrom across the analysis results to emphasize or reject cer-tain hypotheses, but also will introduce high-level featuresthat can offer new interlinking enhancements for an aug-mented viewers’ experience.

Figure 5: Top 10 TREC-Vid concepts detected for two example screenshots. Wrong concepts in the left one are “out-door”,“table”,“computers”, and “clearing”, whereas the rest is correct. In the right one, only three concepts “face”, “bodypart” and “adult” are correct..

Acknowledgements

This work has been partly funded by the European Community’sSeventh Framework Programme (FP7-ICT) under grant agree-ment n◦ 287911 LinkedTV.

References[1] Bay, H., Ess, A., Tuytelaars, T., and Van Gool, L.

(2008). Speeded-up robust features (surf). Comput. Vis.Image Underst., 110(3):346–359.

[2] Fischler, M. A. and Bolles, R. C. (1981). Random sam-ple consensus: a paradigm for model fitting with appli-cations to image analysis and automated cartography.Commun. ACM, 24(6):381–395.

[3] Liu, X. and Huet, B. (2010). Concept detector refine-ment using social videos. In VLS-MCMR'10,International workshop on Very-large-scale multime-dia corpus, mining and retrieval, October 29, 2010,Firenze, Italy, Firenze, ITALY.

[4] Moumtzidou, A., Sidiropoulos, P., Vrochidis, S.,Gkalelis, N., Nikolopoulos, S., Mezaris, V., Kompat-siaris, I., and Patras, I. (2011). ITI-CERTH participa-tion to TRECVID 2011. In TRECVID 2011 Workshop,Gaithersburg, MD, USA.

[5] Nielsen (2009). Three screen report. Technical report,Nielsen Company.

[6] Reynolds, D., Quatieri, T., and Dunn, R. (2000).Speaker Verification Using Adapted Gaussian MixtureModels. Digital Signal Processing, 10:19–41.

[7] Robertson, S. E. and Walker, S. (1994). Some sim-ple effective approximations to the 2-poisson modelfor probabilistic weighted retrieval. In Proceedingsof the 17th annual international ACM SIGIR confer-

ence on Research and development in information re-trieval, SIGIR ’94, pages 232–241, New York, NY,USA. Springer-Verlag New York, Inc.

[8] Schmidt, T. and Wörner, K. (2009). EXMARaLDA –Creating, analysing and sharing spoken language cor-pora for pragmatic research. Pragmatics, 19:4:565–582.

[9] Schneider, D., Schon, J., and Eickeler, S. (2008).Towards Large Scale Vocabulary Independent SpokenTerm Detection: Advances in the Fraunhofer IAIS Au-diomining System. In Proc. SIGIR, Singapore.

[10] Sidiropoulos, P., Mezaris, V., Kompatsiaris, I.,Meinedo, H., Bugalho, M., and Trancoso, I. (2011).Temporal video segmentation to scenes using high-levelaudiovisual features. Circuits and Systems for VideoTechnology, IEEE Transactions on, 21(8):1163 –1177.

[11] Stein, D., Apostolidis, E., Mezaris, V.,de Abreu Pereira, N., and Müller, J. (2012). Semi-automatic video analysis for linking television to theweb. In Proc. FutureTV Workshop, pages 1–8, Berlin,Germany. to appear.

[12] Tsamoura, E., Mezaris, V., and Kompatsiaris, I.(2008). Gradual transition detection using color co-herence and other criteria in a video shot meta-segmentation framework. In Image Processing, 2008.ICIP 2008. 15th IEEE International Conference on,pages 45 –48.

[13] Yahoo! and Nielsen (2010). Mobile shopping frame-work – the role of mobile devices in the shopping pro-cess. Technical report, Yahoo! and The Nielsen Com-pany.