View
4.823
Download
0
Category
Preview:
Citation preview
InteractiveSearchinVideo&Lifelog Repositories
KlausSchoeffmann,PhDKlagenfurtUniversityInstituteof InformationTechnologyKlagenfurt,Austria
FrankHopfgartner,PhDUniversityof GlasgowSchoolof HumanitiesGlasgow,UK
Interactive Search in Video & Lifelog Repositories
• Part1:InteractiveVideoSearchØ Searchinvideocontent:motivationandchallengesØ Automaticvideoretrievalvs.interactivevideosearchØ Toolsforinteractivesearch
§ Browsing,Navigation,Visualization,Similarity&Sketch-basedSearchØ EvaluationofIVSTools
§ TRECVID,VideoBrowserShowdown(VBS)
Shortbreak
• Part2:LifeloggingØ QuantifiedSelfØ LifelogrepositoriesØ LifeloggingtechniquesØ Interactivevisualization
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 2
Video Everywhere
• UbiquitoususeofvideosnowadaysØEntertainment andcommercialsØSocialgaming(screencasts)ØPersonalvideos(family,kids,…)ØSportsdocumentationandanalysis(e.g.,GoPro)ØProductusageinstructions(e.g.,furniture)ØSurveillance(buildings,places,street,…)ØHealthcareandmedicalscience(endoscopicprocedures)ØLifelogging
• Enormousamountofdata,challengingtosearch!
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 4
Video – The Ultimate Media?
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 5
[MaryMeeker,LiangWu,InternetTrends,D11Conference,May,2013]
Asof 2014,everyminute 300hours ofvideo are uploaded
to YouTube!
Video Cameras
• IncreasinglypowerfulØThesedaysyoucanrecord4Kcontentwithyourmobile!ØVideosensorsuseauto-focus,objecttracking,colorcorrection,and imagestabilization
ØStoragespacenotabigproblem§ Currentsmartphoneshave128GBofmemory§ NASdevicescheaplyavailable
ØNetworkbandwidthalsodramaticallyincreasedoveryears§ Videostreamingonthegoissimpleandcommon§ LTEconnectionsprovide30Mbit/sandevenmuchmore!
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 6
7KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
[MaryMeeker,LiangWu,InternetTrends,D11Conference,May,2013]
Challenge: Finding Content
• EvenwithretrievaltoolsstillchallengingtofindcontentlaterØEspeciallyifnotpubliclyavailable(andpopular+annotated)ØManyproblemswithquerying,inparticularfornon-experts
• Ultimategoal:makesearchaseffectiveasfortextØQuicklyfindrelevantcontentØComparetointeractivityofatextbook
§ Index,ToC,listoffigures/tables,etc.§ Change,extend,copy,bookmark,highlight,etc.
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 8
Example Scenario
10
Why? (e.g.,showtosomeone,includeineditedvideo,findsomeinformation,extractimage,etc.)
Youwanttofindthisvideoclipinyourcollection:
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Large Video Collection
11
IACCdataset,asusedforTRECVID:146,788shots
(~9,000videos)
Page123….383940
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
How a Novice Would Solve ThisNovice users typically employ a file browser and a simple video player!
VCRinthe1970sprovidedasimilarfunctionality!
12
?KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Fileexplorerandvideoplayer
13
Factor>1Mio!
[en.wikipedia.org]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
How a Novice Would Solve ThisNovice users typically employ a file browser and a simple video player!
VCRinthe1970sprovidedasimilarfunctionality!
14KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Fileexplorerandvideoplayer
• Videoretrievaltoolwithcontentanalysisandsearch• Queryby
ØText,Concept,Example
• AutomaticsearchØContent-baseddatasuchas:
§ Text (e.g.,metadata,ASR,OCR,transcripts,…)
§ Globalfeatures(e.g.,color,texture,motion)
§ Localfeaturesandconcepts(e.g.,VLAD,BoVW,…)
ØRankedresultlist
15
IBMTRECVID2007VideoRetrieval System[1]
How a Retrieval Expert Would Solve This
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
16
Content-basedFeature
ExampleImage
Text
Rankedlistofshots
InIACCabout5800pages.L
TemporalContext
[Heesch,D.,Howarth,P.,Magalhaes,J.,May,A.,Pickering,M.,Yavlinsky,A.,&Rüger,S.(2004,November).Videoretrieval using search and browsing.InTRECVideoRetrieval EvaluationOnlineProceedings.]
How a Retrieval Expert Would Solve This
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
17
Thiswas10yearsago,whataboutstate-of-the-art?
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
A More Recent Video Retrieval Tool
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 18
[A.Moumtzidou etal.,“VERGE:AMultimodalInteractiveVideoSearchEngine”,Proc.of the 21stInternationalConferenceonMultiMedia Modeling(MMM2015),Sydney,2015]
kNN Similarity searchbased onVLADvectors
Concept detection with SVMandfive local descriptors (SIFT,SURF,
ORB,...)and PCAor CNNs
Hierarchicalkeyframe clustering
22
Concept-basedsearchstillfarfromoptimal(evenwithCNNs)!Evenwithperfectresults,whowouldbrowseafew1000shots?
ShortcomingsoftheQuery-and-Browse Approach
23KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Common Video Retrieval Approach
WorkswellifØuserscanproperlyexpresstheirneeds.Øcontentfeaturescansufficientlydescribevisualcontent.Øcomputervisioncanaccuratelydetectsemantics.
24
Content-basedSearch
Ranked Results
Unfortunately,inpracticetheseassumptionsdonothold.
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
ØContent-basedfeatures§ Howtounderstandsemanticsfrompixels? SemanticGap
Bothimagesshowbearsinfrontofalandscape.
25KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Mind the Gap!
ØDatabaseaffinityofconceptclassifiersØLowperformanceinbroaddomain
P(k) Precisionatlevelk(afterkresults)rel(k) definesifkth retrieveddocumentisrelevant
PerformanceGap
26
TRECVID2015SemanticIndexing(60concepts):median“inferredaverageprecision”(infAP)=0.24
Inotherwords:morethan75%
ofresultsarewrong!
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Mind the Gap!
Ø Query-by-concept§ Whichconcepttouse?Choosefromalonglistofresults…
Ø Query-by-example§ Typicallynoperfectexampleavailable.
Ø Query-by-sketch§ UsersarenoartistsJ (seealsonextslide)
Ø Query-by-text§ Howtodescribeadesiredimagebytext?
UsabilityGap
27
Apicturetellsa1000words.
bymarfis75
Howtodescribeadesiredvideoclipbytext???KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Mind the Gap!
Needs More Focus on the User (Interface)!
Ø Insomesituationsuserscannotformulateaquery§ à provideexploratorysearchfeatures!§ Forexample:browsing,filtering,similaritysearch
ØUsersexpectgoodresults(onfirstpage!)§ à Userelevancefeedback/activelearning insteadoflonglists!
ØVideosaredynamic§ Staticthumbnailsarenotinformative§ Esp.trueforlongshotsandself-similarcontent§ à skimsandvisualsummaries(“smartplayback”)§ à sophisticatednavigation&contentstructurevisualization
ØShotshaveatemporalcontextØGridinterfacesarenotalwaysthebestchoice
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 28
UsabilityGap
Interactive Video Search
30
• HCIcommunity• Methodsforinteractivesearch• Humancomputation• Nocontentunderstandingbutsimple
• Multimediacommunity• Mostlyautomaticsearch• Retrievalengine• Complicatedtouse
Mismatch
Novices Experts
à CombineHCIwithCVandMIRforbettersearchtools
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
User-Centric Exploratory Search
• Stronglyintegrateuser intosearchprocessØ AssumeasmartuserØ Givehim/hermorecontroloversearchprocess
§ Inspectsandinteracts§ Selectsmostmeaningfultoolforcurrentneeds,e.g.
• ContentBrowsing/Navigation• ContentVisualizationandSummarization• Ad-hocQuerying(e.g.,bysketch, filtering,ad-hocexample)• Aspect-basedexploration,parallelsearchpaths
Ø Iterative:Search– Inspect– Think– Repeat§ Exploratorysearch(“willknowitwhenIseeit”)§ Insteadof„query-and-browse-results“
31KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Aspects of Interactive Video Search (IVS)
IVS
Navigation &Browsing
DifferentQueryTypes
Dynamics&Convenience
ContentVisualization
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 32
UnderlyingStructure
Abstracts/summaries
Overview(TOC)
Skims
SmartPlayback
Bookmarks
History
TextorConcept
ExampleImage
ExampleClip
(SimilaritySearch)
Sketch
Filter(Spatial&Temporal)
CoarseNavigationFineNavigation
BrowsingSequences/Scenes/Shots
Similarity-BasedArrangements(e.g.,byColor)
Outline
InteractiveVideoSearch(IVS)Tools:ØVideoNavigationØVideoBrowsingØContentVisualizationØSketch-basedSearch
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 33
Improving Navigation
35
e.g.,onYouTubedefaultwindow:
640pixels=frames(25seconds)
Commonseeker-barlimitsnavigationgranularity
[Huerst etal.,ICME2007]
ZoomSlider
Improvements(selected):
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Improving Seeker-Bar Navigation
36
WolfgangHürst,GeorgGötz,andMartinaWelte,“Interactivevideobrowsingonmobiledevices”,inProceedingsofthe15thInternationalConferenceonMultimedia (MULTIMEDIA'07).ACM,pp.247-256,2007
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
ZoomSlider[Huerst etal.,ICME2007]
Improving Navigation
38
e.g.,onYouTubedefaultwindow:
640pixels=frames(25seconds)
Commonseeker-barlimitsnavigationgranularity
[Dragicevic etal.,CHI2008]
DirectManipulation
[Huerst etal.,ICME2007]
ZoomSlider
Improvements(selected):
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Relative Flow DraggingBackground Stabilization
39
PierreDragicevic,GonzaloRamos,Jacobo Bibliowitcz,DerekNowrouzezahrai,Ravin Balakrishnan,andKaranSingh.“Videobrowsingbydirectmanipulation”,inProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems(CHI'08).ACM,pp.237-246,2008
Videobrowsingbydirectmanipulation/relativeflowdragging
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Relative Flow Dragging
• EvaluationwithauserstudyØ 16participants(18-44yearsold)Ø Directcomparisontoseeker-barnavigationØ Navigationtasks,2videos(ladybug,cars)
§ “FindthepositionwheretheladybugpassesovermarkerX”§ “FindthemomentwhencarXstartsmoving”
Ø Flowdraggingsignificantlyfaster(RM-ANOVA)byatleast250%(alsosignificantlylesserrors)
40
PierreDragicevic,GonzaloRamos,Jacobo Bibliowitcz,DerekNowrouzezahrai,Ravin Balakrishnan,andKaranSingh.“Videobrowsingbydirectmanipulation”,inProceedingsoftheSIGCHIConferenceonHumanFactorsinComputingSystems(CHI'08).ACM,pp.237-246,2008
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Scrubbing Wheel
• RequirementsØSimpleandeffectivenavigationontouchscreens
ØEfficientnavigationthatallowsforcontentsearchinbothshortandlongvideos
• IdeaØ improvenavigationbyusingacircularnavigationarea
Ø inspiredbyAppleiPod(c)device
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 42
KlausSchoeffmann andLukasBurgstaller,“ScrubbingWheel:AnInteractionConcepttoImproveVideoContentNavigationonDeviceswithTouchscreens“,inProceedingsoftheIEEEInternationalSymposiumonMultimedia2015(ISM2015),Miami,FL,USA,2015,pp.351-356
Scrubbing Wheel Implementation (iOS)
43KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
46
Video Browsing
[F.Arman,R.Depommier,A.Hsu,andM-Y.Chiu,Content-basedBrowsingofVideoSequences,inProc.ofACMInternationalConferenceonMultimedia,1994,pp.97-103]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Video Browser for the Digital Native
47
[Adams, Brett, Stewart Greenhill, and Svetha Venkatesh. "Towards a video browser for the digital native." Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on. IEEE, 2012.]
“TemporalSemanticCompression”basedontempofunctionandshotpopularity(insight)
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Video Browser for the Digital Native
• Userstudywith8participantsØ Testconfigurationelementsbytwotasks(afterpresentation+5minutestraining)§ (i)Browseafamiliarmovietofindscenesyouremember§ (ii)Browseanunfamiliarmovietogetafeelforitsstoryorstructure
Ø QuestionnairewithLikert-scaleratings
48
[Adams, Brett, Stewart Greenhill, and Svetha Venkatesh. "Towards a video browser for the digital native." Multimedia and Expo Workshops (ICMEW), 2012 IEEE International Conference on. IEEE, 2012.]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
The Video Explorer
49
[Schoeffmann,K.,Taschwer,M.,&Boeszoermenyi,L.(2010,February).Thevideo explorer:atool for navigation and searching within asingle video based onfastcontent analysis.InProceedings of the first annualACMSIGMMconference onMultimediasystems (pp.247-258).ACM.]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Interactive Navigation Summaries
Allowsausertoquicklyidentifysimilar/repeatingscenes
50
[Schoeffmann,K.,&Boeszoermenyi,L.(2009,June).Videobrowsing using interactive navigation summaries.InContent-Based MultimediaIndexing,2009.CBMI'09.Seventh Int.Workshop on (pp.243-248).IEEE.]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Motion Layout: Direction + Intensity
MotionVector (µ)classification intoMotionhistogram with K=12
equidistant motion directions (bins)Mappingto Hue channel
51
[Schoeffmann,K.,Lux,M.,Taschwer,M.,&Boeszoermenyi,L.(2009,June).Visualization of video motion incontext of video browsing.InMultimediaand Expo,2009.ICME2009.IEEEInt.Conf.on (pp.658-661).IEEE.]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
52
[Schoeffmann,K.,Lux,M.,Taschwer,M.,&Boeszoermenyi,L.(2009,June).Visualization of video motion incontext of video browsing.InMultimediaand Expo,2009.ICME2009.IEEEInt.Conf.on (pp.658-661).IEEE.]
Similarity Search (SOI) with Motion Layout
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
• SOISearchØ Motion-basedsearchbyexamplesequence
§ UsingMotionDirection histogramDb
§ User-selectedsequence
Ø Findmostsimilarsequences§ Computedistance toanypossibleseq. ofsamelength§ Matchifbelowspec.threshold
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 53
MotionLayout(Db)
Match1 Match2 Match3
frame1 framen
Similarity Search (SOI) with Motion Layout
Region-of-Interest(ROI)SearchØ Userselectsspatialregion-of-interestØ Onsearch
§ ComputeEuclidiandistance offrameFtoeveryotherframe f (acc.toselectedregion)
§ Basedoncolorlayout descriptor
…
frameF
frame1 framek framen
User-selectedregion(I)
…
d(F,1)=350 d(F,k)=8 d(F,n)=400
54
[Schoeffmann,K.,Taschwer,M.,&Boeszoermenyi,L.(2010,February).Thevideo explorer:atool for navigation and searching within asingle video based onfastcontent analysis.InProceedings of the first annualACMSIGMMconference onMultimediasystems (pp.247-258).ACM.]
Similarity Search (ROI) with Color Layout
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
55
[Schoeffmann,K.,Taschwer,M.,&Boeszoermenyi,L.(2010,February).Thevideo explorer:atool for navigation and searching within asingle video based onfastcontent analysis.InProceedings of the first annualACMSIGMMconference onMultimediasystems (pp.247-258).ACM.]
Similarity Search (ROI) with Color Layout
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
The ForkBrowser
• Thread:linkedsequenceofshotsinaspecifiedorderØ Queryresults,visualsimilarity,semanticsimilarity,textualsimilaritytime,…
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 56
[DeRooij,Ork,CeesGMSnoek,andMarcelWorring."Balancingthreadbasednavigationfortargetedvideosearch."Proceedingsofthe2008internationalconferenceonContent-basedimageandvideoretrieval(CIVR).ACM,2008.]
IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
DemoVideo
KlausSchoeffmann 58
Goal:improvetwo-handeduse
The ThumbBrowser
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 59
[MarcoHudelist,KlausSchoeffmann,LaszloBöszörmenyi.“MobileVideoBrowsingwiththeThumbBrowser”,Proc.oftheInternationalConferenceonMultimedia,2013,pp.405-406]
Grid Interfaces Aren‘t Enough!
• ManyvideoretrievalsystemsuseaGridinterface!?
Moreover,agridinterfacedoesnotallowforfasthumanvisualsearch(seelater)!
61
Arankedlistofresultsdoesnotconveythetemporalcontentstructure!• Towhichvideodoesashotbelongto?• Whatisthesequenceofshots?• Howlongisashot/scene?
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
TableofVideoContent(TOVC)
[Goeau etal.,ICME2007]
62
Squeeze/FisheyeRapidVisualSerialPresentation(RSVP)
Improving Visualizationaka “Video Surrogates”
[Wildemuth etal.,2003]
[Wittenburg etal.,2005]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
63
VideoTree[Jansenetal.,CBMI2008]
However,outperformedbysimple“gridofkeyframes”intermsofsearchtime.
Similarconceptproposedlater[Girgensohn etal.,ICMR2011]
• Split-basedclustering algorithmwithcolorcorrelograms.
• Treenotdirectlyshown totheuser(onlyonelevel).
Improving Visualizationaka “Video Surrogates”
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Hierarchical Video BrowsingAnother Tree-based Approach
FrontalView TopView
From:[Schoeffmann andDelFabro,2011]
64
• Goal:improvecontentoverview• Nocontentanalysis(justuniformsamplingofframes)
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
3D Ring Instead of Grid!
• UtilizationofscreenrealestateØ LargesetofimagesØ Minorocclusion,slightdistortion
• IntuitiveinteractionØ Rotateandzoom
• Content-based sorting• “Pop-outimages”(intheback)• Furtheradvantages
Ø Immediatelycontinueonmiss,scaling
66
Klaus Schoeffmann, David Ahlström, and Marco Andrea Hudelist, “3-D Interfaces to Improve the Performance of Visual Known-Item Search“, in IEEE Transactions on Multimedia, Vol. 16, No. 7, November, 2014, pp. 1942-1951.
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
3D Ring Interface - Perspectives
PreferredDesignacc.touserstudy
25%Vertical66%Horizontal 8.3%Frontal
67
Klaus Schoeffmann, David Ahlström, and Marco Andrea Hudelist, “3-D Interfaces to Improve the Performance of Visual Known-Item Search“, in IEEE Transactions on Multimedia, Vol. 16, No. 7, November, 2014, pp. 1942-1951.
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
3Dinterface significantly faster than grid by 12.7%
User Study: Grid vs. Ring (both sorted)150 images, 12 participants, 1440 trials
68
Klaus Schoeffmann, David Ahlström, and Marco Andrea Hudelist, “3-D Interfaces to Improve the Performance of Visual Known-Item Search“, in IEEE Transactions on Multimedia, Vol. 16, No. 7, November, 2014, pp. 1942-1951.
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Extension: Multiple Rings with Vertical Scrolling
69
KlausSchoeffmann.2014.TheStack-of-RingsInterfacefor Large-Scale ImageBrowsing onMobileTouchDevices.InProc.of the ACMInt.ConferenceonMultimedia(MM'14).ACM,NewYork,NY,USA,1097-1100.
Significantly faster search (by about 48%)than common image browser oniPad!
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
• Colorsketchesmappedtofeaturesignatures
• Matchedtothoseofkeyframes
71
1. Samplingkeypoints2. Descriptionthroughlocation(x,y),
CIELab,contrastandentropyofsurroundingpixels
3. k-meansclustering
Feature Signatures
[Kruliš,M.,Lokoč,J.and Skopal,T.(2013).Efficient Extraction of FeatureSignatures Using Multi-GPUArchitecture.SpringerBerlinHeidelberg,LNCS7733,pp.446-456.]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Feature Signature-Based Video Browser
72
ColorSketch(Signature)
Player
WinnerofVideoBrowserShowdown2014+2015Downloaddemoat:http://siret.ms.mff.cuni.cz/lokoc/vbs.zip
2nd ColorSketch(optional)
[Lokoč,J.,Blažek,A.,&Skopal,T.(2014,January).Signature-Based VideoBrowser.InMultiMedia Modeling (pp.415-418).SpringerInternationalPublishing.]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Compactvisualization
Simplecolor-positionsketch
Negativeexample
Matchedkey-frames
Timeto2nd sketch
2nd optionalsketch
Interactive-navigationsummaryOndemandneighborhoodexpansion
[Slide:AdamBlazeketal.(siret researchgroup,CzechRepublic)]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 73
Compact Visualization to Save Space
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 74
[CourtesyofJakubLokoc etal.]
Another Example of a Sketch-Based Browser
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 75
[KaiUweBarthel,Nico Hezel,Radek Mackowiak.Navigatingagraphofscenesforexploringlargevideocollections,inProc.of22ndInternationalConferenceonMultiMedia Modeling(MMM2016),LectureNotesinComputerScience(LNCS),Vol.tbd,SpringerInternationalPublishing,2016,pp.1-7]
WinnerofVideoBrowserShowdown2016
User Studies with Significance Tests!
• Manyinterfacesproposedwithoutproperevaluation• InterfaceAbetterthaninterfaceB?à comparativeuserstudyneeded!
Ø Performsearchtasksinexactlythesamesetting(data,environment,etc.)
Ø Loggingofinteractionbehaviorandtasksolvetime
Ø QuestionnaireaboutsubjectiveworkloadsØ Statisticalanalysiswithpropertests(e.g.,t-test,ANOVA,Wilcoxonsigned-rank,etc.)
• Usersimulations?• Evaluationcompetitions
Ø SamedatasetØ ComparativeevaluationØ TRECVID,MediaEval,VideoBrowserShowdown
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 78
Video Browser Showdown (VBS)
• AnnualperformanceevaluationcompetitionØ LiveevaluationofsearchperformanceØ SpecialsessionatInt.ConferenceonMultiMedia Modeling(MMM)Ø Demonstratesandevaluatesstate-of-the-artinteractivevideosearchtoolsØ IdeainfluencedbyVideOlympics (Snoek etal.,IEEEMultimedia2008)
• FocusØ Known-itemSearchtasks
§ Targetclipsarepresentedonsite§ Teamssearchinshareddataset
Ø Highlyinteractivesearch§ Shouldpushresearchoninterfaces
andinteraction/navigationØ Experts andNovices
§ Easy-to-usetoolsandmethodsØ Ad-HocVideoSearch(TRECVIDAVS)tasksstartingfrom2017
79
http://videobrowsershowdown.org/
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Video Browser Showdown (VBS)
• Liveevaluation/scoringthroughVBSServer• Score(s)[0-100]fortaski andteamk isbasedon
ØSolvetime(t)ØPenalty(p)basedonnumberofsubmissions(m)
80
Maximumsolvetime(Tmax)typically5minutes
[Schoeffmann,K.,Ahlström,D.,Bailer,W.,Cobârzan,C.,Hopfgartner,F.,McGuinness,K.,...&Weiss,W.(2013).TheVideoBrowserShowdown:aliveevaluation of interactive video search tools.InternationalJournalof MultimediaInformationRetrieval,1-15.]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Correctbutsubmittedlaterthanfirstteam Penaltyduetotoomany
wrongsubmissions
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 81
Video Browser Showdown 2016
• Searchinmid-sizedvideocollectionsØ Originallyonlysinglevideosearch
• TwodifferentkindofKIStasks:Ø Visual:visualpresentationofa30stargetclipØ Textual:textualdescriptionofa30stargetclip
• SharedvideodatafromBBCØ 2016:441videofiles,about320.000shots(250hours)
[Schoeffmann,Klaus."Auser-centricmediaretrievalcompetition:Thevideobrowsershowdown2012-2014."MultiMedia,IEEE 21.4(2014):8-13.]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 82
Visual Task Example (2016)
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 83
Textual Task Example (2016)
“Stevecuttingadrawingintohisblockofwood.Youcanseehishandandacutterandflowersymbols.”
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016 84
85KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
2012:Klagenfurt11 teams
2013:Huangshan6teams
2014:Dublin7teams
2015:Sydney9teams
2016:Miami9teams
VBS2017:January4,2017,Reykjavik,Iceland(MMM2017)http://www.videobrowsershowdown.org/
Winner 2014 and 2015(2014: single video and collection search, 2015: collection only)
86
ColorSketch(Signature)
Player
2nd ColorSketch(optional)
[Lokoč,J.,Blažek,A.,&Skopal,T.(2014,January).Signature-Based VideoBrowser.InMultiMedia Modeling (pp.415-418).SpringerInternationalPublishing.]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Video Browser Showdown 2015Two other examples of the 9tools (collection search only)
87
Moumtzidou,A.,Avgerinakis,K.,Apostolidis,E.,Markatopoulou,F.,Apostolidis,K.,Mironidis,T.,...&Patras,I.(2015,January).VERGE:AMultimodalInteractiveVideoSearchEngine.InMultiMedia Modeling(pp.249-254).SpringerInternationalPublishing.
• Shotandscenedetection• HLF(Concepts)with
SIFT/SURFandVLAD• Similaritysearch
• Uniformsampledframes• Humancomputation
Hürst,W.,vandeWerken,R.,&Hoet,M.(2015,January).AStoryboard-BasedInterfacefor MobileVideoBrowsing.InMultiMedia Modeling (pp.261-265).SpringerInternationalPublishing.
3rd place
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
Human vs. Machine
• UtrechtUniversity@VBS2015Ø WolfgangHuerst etal.,TheNetherlandsØ Strongexperience inHCI
• FeaturesØ Uniformly sampled thumbs(1second distance)
Ø Huge storyboard ontabletØ Vertical scrolling,paging
88
625thumbnails inone screen
[Hürst,W.,vandeWerken,R.,&Hoet,M.(2015,January).AStoryboard-Based Interfacefor MobileVideoBrowsing.InMultiMedia Modeling (pp.261-265).SpringerInternationalPublishing.]
KlausSchoeffmann IEEEInternationalConferenceonMultimedia&Expo(ICME)2016
FrankHopfgartnerSchoolofHumanities
Universityof Glasgow,UK
Tutorial:InteractiveSearchinVideo&LifelogRepositoriesPart2:TheQuantifiedSelfandLifelogging
IEEEInternationalConferenceonMultimediaandExpo(ICME)2016
Afewwordsaboutme
Research on Multimedia Analysis, Quantified Self, Lifelogging
Lecturer(AssistantProfessor)inInformationStudies(UGlasgow)
PhDinComputingScience(UniversityofGlasgow)
Past:VariouspositionsinBerlin(TUB),Dublin(DCU),Berkeley(ICSI),andLondon(QMUL)
What is The Quantified Self?
Self-trackingisalsoreferredtoaslifelogging,self-analysis,orself-hacking.
Memex
Bush,Vannevar."AsWeMayThink."TheAtlanticMonthly.July1945.
ImagesofM
emex:http://trevor.sm
ith.nam
e/mem
ex/
MyLifeBits
• GordonBell(Microsoft)digitizedhislife:ØBookswrittenØPersonaldocumentsØPhotosØPosters,paintings,photoofthings
ØHomemoviesandvideosØCDcollectionØPCfilesØ…
Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton 2009
http://research.microsoft.com/en-us/projects/mylifebits/
MyLifeBits
Slidefrom:G.Bell.ChallengesinUsingLifetimePersonalInformationStoresbasedonMyLifeBits.PresentationatAlpbach Forumon26August2004.
Creating Personal Lifelog Repositories
Alifelogrepositoryconsistsofheterogeneousdatarecordedusingmanydifferentsensors.
In this tutorial, we will…
• getanintroductionintothecreationoflifelogrepositories
• understandthemajorchallengesofcreatinglifelogrepositories
• discusshowtoevaluatelifeloggingtechniques.
So what are the challenges?
Thechallengesarehowtosensetheperson,capturetheiractions,theirlifeandmakeitaccessibleusing
appropriategraphicaluserinterfaces,search/recommendationenginesandvisual/auralfeedback.Further,exploitingthelifelog toidentify
contextforadaptiveinformationservices.
Research communities
Multimedia
ACMMultimedia
IEEEICME
MultimediaModeling
HCI
ACMCHI
AugmentedHuman
ACMUbiComp
MachineLearning
ICML
KDD
ECML
Recording my media consumption
Brusilovsky,P.andKobsa,AlfredandNejdl,Wolfgang.“TheAdaptiveWeb:MethodsandStrategiesofWebPersonalization."LectureNotesinComputerScience,SpringerVerlag,2007.
Recording my communicationIm
age:http://www.wire
d.co.uk/news/archive/2013-
06/10/sim
ple-guide-to-prism/viewgallery/304880
(Automatically) recording who I meet
• Inferred,weightedfriendshipnetworkvs.reported,discretefriendshipnetwork.
Eagle,NathanandPentland,Alex(Sandy)andLazer,David.“Inferringfriendshipnetworkstructurebyusingmobilephonedata."ProceedingsoftheNationalAcademyofSciencesoftheUnitedStatesofAmerica,106(36):15274-15278,2009.
Recording what I eat
Aizawa,Kiyoharu,Maruyama,Yutu,Li,He,andMorikawa,Chamin.“FoodBalanceEstimationbyUsingPersonalDietrary TendenciesinaMultimediaFoodLog."IEEETransactionsonMultimedia,15(8):2176-2185,2013.
SemanticGap
http://foodlog.jp/
http://mealsnap.com/
Recording what I see
"LifeGlogging cameras1998200420062013labeled"byGlogger - Ownwork.LicensedunderCCBY-SA3.0viaCommons-https://commons.wikimedia.org/wiki/File:LifeGlogging_cameras_1998_2004_2006_2013_labeled.jpg#/media/File:LifeGlogging_cameras_1998_2004_2006_2013_labeled.jpg
Big Data
CathalGurrin,AlanF.SmeatonandAidenR.Doherty(2014),"LifeLogging:PersonalBigData",FoundationsandTrends®inInformationRetrieval:Vol.8:No.1,pp1-125.
Event Segmentation & Annotation
• Segment5,500photosperdayintoasetofeventsØ SimilartoSBDindigitalvideoprocessingØ Weemployvisualfeaturesandoutputofon-devicesensors
MultipleEvents
Finishingworkinthelab
Atthebusstop ChattingatSkylon Hotellobby Movingtoaroom
Teatime Onthewaybackhome
EventSegmentation
Summarization
Slide:CathalGurrin
Context is key
• Contextcueshelpustoremember(Naaman etal.)
• Contextinlifeloggingdata:Ø Location,bluetooth,time,date,…
Ø DerivedKnowledge(e.g.activities)
• Approaches:Ø Combinecuesfromdifferentsources
Ø Performcontentanalysistoidentifyobjects,people,events…
Ø Annotatelifelogsinformofnarrativetext
Mor Naaman,SusumuHarada,QianYing Wang,HectorGarcia-Molina,AndreasPaepcke:Contextdataingeo-referenceddigitalphotocollections.ACMMultimedia2004:196-203
Visual Feature Extraction
Ø Steeringwheel(72%)Ø Shopping(75%)Ø Insideofvehiclewhennotdriving(airplane,taxi,car,
bus)(60%)Ø Toilet/Bathroom(58%)Ø GivingPresentation/Teaching(29%)Ø ViewofHorizon(23%)Ø Door(62%)Ø Staircase(48%)Ø Hands(68%)Ø Holdingacup/glass(35%)Ø Holdingamobilephone(39%)Ø Eatingfood(41%)Ø Screen(computer/laptop/tv)(78%)Ø Readingpaper/book(58%)Ø Meeting(34%)Ø Road(47%)Ø Vegetation(64%)Ø OfficeScene(72%)Ø Faces(61%)Ø People(45%)Ø Grass(61%)Ø Sky(79%)Ø Tree(63%)
Byrne,Daragh,Doherty,AidenR.,Snoek,CeesG.M.,Jones,GarethJ.F.,Smeaton,AlanF.“Everydayconceptdetectioninvisuallifelogs: validation,relationshipsandtrends."MultimediaToolsandApplications,49(1):119-144,2010.
Non-supervised Event Segmentation
2. Arriving in the office
6. Walking inthe building 12. Leaving
the office
NaLietal.“RandomMatrixEnsemblesofTimeCorrelationMatricestoAnalyze VisualLifelogs."InProc.MultimediaModeling Conference,Dublin,Ireland,pp.400-411,2014.
EventSegmentationbasedontheextractionoflowlevelfeaturesandcomputationofsemanticconceptsrequiresknowledgeaboutdataset.
Alternative:Highlight“significantevents”byperformingtimeseriesanalysis
People access memory for five reasons
Sellen,AbigailandWhittaker,Steve.“BeyondTotalCapture:AConstructiveCritiqueofLifelogging."CommunicationsoftheACM,53(5):70-77,2010.
•Relivingpastexperiencesforvariousreasons
Recollecting
•Story-tellingorsharinglifeexperienceswithothers
Reminiscing
•Findspecificinformationsuchasanaddress,oradocument
Retrieving
•Gaininginsights(QuantifiedSelf)
Reflecting
•Planningfutureactivities.
Remembering
Quantified Self
P. Kostopoulos. Stress Detection using Smartphone Data. In Proc. HealthWear’16, Budapest, Hungary, 2016
Reflecting
• Reflecting isaformofquantifiedself-analysisoverthelifearchivedatatodiscoverknowledgeandinsightsthatmaynotbeimmediatelyobvious.
• Example: NickFeltronAnnualReports
Image:©NickFeltron.
MyLifeBits
Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton 2009
MyLifeBits
Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton 2009
MyLifeBits
Gordon Bell and Jim Gemmell. Total Recall: How the E-Memory Revolution will change everything, New York, Dutton 2009
Interactive visualization
Hwang,Keum-SungandCho,Sung-Bae.“ALifelogbrowserforvisualizationandsearchofmobileeveryday-life."MobileInformationSystems,10(2013):243-258.Jeon,JaeHo andYeon,Jongheum andLee,Sang-gooandSeo,Jinwook.“ExploratoryVisualizationofSmartphone-basedLifeloggingDatausingSmartRealityTestbed.”InProc.BigDataandSmartComputing,pp.29-33,2014
Virtual reality
“BadTripisanimmersivevirtualrealityinstallation[…]thatenablespeopletonavigatethecreator'smindusingagamecontroller.SinceNovember2011,everymomentsofhislifehasbeendocumentedbyavideocameramountedonglasses,producinganexpandingdatabaseofdigitalizedvisualmemories.Using customvirtualrealitysoftware,hecreatedavirtualmindscapewherepeoplecouldnavigate,andexperiencehismemoriesanddreams.”
Souce:http://www.kwanalan.com
Art installations
Kelly,PhilipandDoherty,AidenR.andSmeaton,AlanF.andGurrin,CathalandO’Connor,NoelE.“TheColourofLife:NovelVisualisationsofPopulationLifestyles."InProc.ACMMultimedia,pp.1063-1066,2010.
Image:Cou
rtesyofC.G
urrin
Displaying photo stream
Image:http://thenextweb.com/gadgets/2013/07/29/autographer-review-we-put-this-615-wearable-life-logging-camera-to-the-test/
Browsing in the Living Room
• Controlwithasuiteofgestures:ØNext/previouseventØNext/previousimageØNext/previousday,week,…
• Possibilityofpivotviewacrossmultipleaxes,e.g.,people,locations,…
Gurrin,CathalandLee,Hyowon andCaprani,NiamhandZheng,Zhenxing andO’Connor,NoelandCarthy,Denise.“BrowsingLargePersonalMultimediaArchivesinaLean-backEnvironment."InProc.MultimediaModeling Conference,pp.98-109,2010.
SenseCam Viewer
Doherty,AidenR.,Moulin,ChrisJ.A.,andSmeaton,AlanF.(2011)AutomaticallyAssistingHumanMemory:ASenseCam Browser.,Memory:SpecialIssueonSenseCam:TheFutureofEverydayResearch?TaylorandFrancis,19(7),785-795
Browsing Interface
Lee,Hyowon,Smeaton,AlanF.,O’Connor,NoelE.,Jones,GarethJ.F.,Blighe,Michael,Byrne,Daragh,Doherty,AidenR.,Gurrin,Cathal.“ConstructingaSenseCam visualdiaryasamediaprocess."MultimediaSystems,14(6):341-349,2008.
Lifelog Insight Tool
AaronDuane,RashmiGupta,LitingZhou,andCathalGurrin.“VisualInsightsfromPersonalLifelogs."InProc.NTCIR12,2016.
Highlighting Key Moments
Hopfgartner,F.andYang,YangandZhou,Lijuan andGurrin,Cathal.“UserInteractionTemplatesfortheDesignofLifeloggingSystems."InSemanticModelsforAdaptiveInteractiveSystems.Chapter10,pp.187-204,2013.
Lifelog Moment Retrieval“FindthemomentswhenI’mdrinkingcoffeeinfrontofmylaptop”
G.DeOliveiraBarra,A.CartasAyala,M.Bolanos,M.Dimiccoli,X.Giro-i-Nieto,P.Radeva.“LEMoRe:ALifelogEngineforMomentsRetrievalattheNTCIR-LifelogLSATTask."InProc.NTCIR12,2016.
Reminiscing
• Reminiscing isaboutstory-tellingorsharinglifeexperienceswithothers.
Image:CourtesyofC.Gurrin
Open Research Questions
• Multimediasummarisation• Handlingheterogeneousdatastreams• Visualisation oflifelogs• RetrievalandRecommendation• …
NTCIR
• WorkshopseriesfocusingonresearchonInformationAccess technologies(informationretrieval,questionanswering,textsummarisation,etc)
• InitiallysponsoredbyJapanSocietyforPromotionofScience (JSPS)
• Organisedsince1997inan18-monthscycle• NTCIR-12:January2015– June2016
NIITestCollectionforIRSystems
NTCIR-12 Tasks
NTCIR-12
§ Secondround:§ Search-IntentMining§ MobileClick§ TemporalInformationAccess§ SpokenQuery&SpokenDocumentRetrieval§ QALabforEntranceExam
§ Firstround:§ MedicalNLPforClinicalDocuments§ PersonalLifelog Access&Retrieval§ ShortTextConversation
Encourageresearchadvancesinorganisingandretrievingfromlifelogdata.
LifeLog @ NTCIR-12
C.Gurrin,H.Joho,F.Hopfgartner,L.Zhou,R.Albatal.OverviewofNTCIR-12LifelogTask.InProc.NTCIR-12,Tokyo,Japan,2016
Multimodal dataset with information needs
Createdbythreeindividualsover
10+days
TESTCOLLECTION
§ 18.18GB§ 88,124images§ Accompanyingoutputof
1,000concepts (825MB)§ Dataprocessedpre-release
(removalofpersonalcontent;faceblurring,translationofconcepts)
§ Detaileduserqueriesandjudgmentsgeneratedbythelifeloggingdatagatherers
C.Gurrin,H.Joho,F.Hopfgartner,L.Zhou,R.Albatal.NTCIRLifelog:TheFirstTestCollectionforLifelogResearch.InProc. SIGIR’16,toappear.
Tasks
Evaluatedifferentmethodsofretrievalandaccess.
T1:LIFELOGSEMAN
TICAC
CESS(LSAT) § Modelstheretrievalneed
fromlifelogs(Known-itemSearch)
§ RetrieveNsegmentsthatmatchinformationneed
§ InteractiveorAutomaticparticipation
§ Interactive:Timelimitforfairandcomparativeevaluationinaninteractivesystemwithusers
§ Automatic:Fully-automaticretrievalsystem.Automatedqueryprocessing
T2:LIFELOGINSIGH
T
§ Modelstheneedforreflectionoverlifelogdata
§ Exploratorytask,theaimisto:§ Encouragebroad
participation§ Novelmethodsto
visualizeandexplorelifelogs
§ SamedataasLSATtask§ Presentedviademo/poster
Tasks
Evaluatedifferentmethodsofretrievalandaccess.
T1:LIFELOGSEMAN
TICAC
CESS(LSAT) § Aknownitemsearchtaskto
findmoments§ Automaticandinteractive
(4&1participants)§ 48queries§ Unitofretrievalwasthe
moment§ Anyimagewithina
momentcanbesubmitted
T2:LIFELOGINSIGH
T
§ Modelstheneedforreflectionoverlifelogdata
§ Exploratorytask,theaimisto:§ Encouragebroad
participation§ Novelmethodsto
visualizeandexplorelifelogs
§ SamedataasLSATtask§ Threeparticipants
Example LSAT Topic
Title: TowerBridge
Description: Findthemoment(s)whenIwaslookingatTowerBridgeinLondon
Narrative: Tobeconsideredrelevant,thefullspanofTowerBridgemustbevisible.MomentsofcrossingtheTowerBridgeorshowingsomesubsetofTowerBridge
arenotconsideredrelevant
Example LIT Topics
Title:Whohasamorehealthylifestyle?
Description: Comparethelifestyleofallthreeuserswithinthedimensionofpersonalhealthandwellness
Narrative: Therearemanyaspectstoahealthylifestyle,suchastheamountofexercise,thefoodanddrinkconsumed,environmentalfactors,thelevelofsocialinteractionsandsleeptime.Thistopicisseekingtounderstandwhichoftheuserswouldbeconsideredtobethemosthealthy.Any
dimension(orcombinationofdimensions)ofhealthylifestyleisconsideredacceptableasapointofcomparison.
AaronDuane,RashmiGupta,LitingZhou,andCathalGurrin.“VisualInsightsfromPersonalLifelogs."InProc.NTCIR12,2016.
Task 1: Lifelog Semantic Access
Findthemoment(s)
whereIusemycoffeemachine.
Findthemoment(s)
whereIaminthekitchen
Findthemoment(s)whereIam
playingwithmyphone.
Findthemoment(s)whereIampreparingbreakfast.
http://ntcir-lifelog.computing.dcu.ie/
Task 2: Lifelog Insight Task
ProvideinsightsonthetimeIspendtakingbreakfast.
ProvideinsightsonthetimeI
spenddrivingtowork.
ProvideinsightsonthetimeI
spendreadingapaper.
ProvideinsightsonthetimeIspendworking
onthecomputer.
http://ntcir-lifelog.computing.dcu.ie/
Evaluation (Task 1)
• Automaticrunsassumethattherewasnouserinvolvementinthesearchprocessbeyondspecifyingthequery.Thesearchsystemgeneratesarankedlistofupto100momentsforeachtopicandnotime.
• Interactiverunsassumethatthereisauserinvolvedinthesearchprocessthatgeneratesaqueryandselectswhichmomentsareconsideredcorrectforeachtopic.
Ø 1.Ininteractiveruns,themaximumtimeallowedforanytopicis5minutes
Ø 2.WeusedthetimeelapsedtocalculaterunperformanceatdifferenttimeCut-offs.TheCut-offswereselectedas10s,30s,60s,120s,300s.
• EvaluationMetricsØ MeanAveragePrecision(MAP)Ø NormalisedDiscountedCumulativeGain(NDCG)
http://ntcir-lifelog.computing.dcu.ie/
Shameless advertisement
ConsiderparticipatinginNTCIRLifelog2andpresentyourworkinEuropeor
Japan
http://ntcir-lifelog.computing.dcu.ie/
Recommended