Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Real-Time News Recommendation and Lifelogging: An Introduction to the Evaluation Campaigns NEWSREEL (CLEF) and Lifelog (NTCIR-12)
Frank Hopfgartner @OkapiBM25
A few words about me
Past: Various positions in Berlin (TUB), Dublin (DCU), Berkeley (ICSI), and London (QMUL)
Research on Information Access
Systems
Lecturer in Information Studies (HATII, Glasgow)
PhD in Information Retrieval (University of Glasgow)
Overview
Evaluation Campaigns
Overview NEWSREEL
(CLEF) LifeLog
(NTCIR-12)
Introduction
Career Collaborations Research
affiliated with UC Berkeley
*now: INSIGHT Centre
Smart Information Systems
Collaborations
Recommender Systems
Aggregated Search
Multimedia Analysis
Gamification Large-Scale
Evaluation
Semantic Search
Multimedia Access
Lifelogging
Sensor Analysis Multimedia Retrieval
& Recommendation
User Modelling
User Simulation
Co-Author graph of first 100 publications (2006-2015)
Example: Video Search Engine
[ACM Multimedia’08, ECIR’08, SIVP’08, ACM TOIS’11]
Adapt retrieval results based on •User’s search queries •Relevance feedback
Example: Video Search Engine
[ACM/Springer Multimedia Systems (2010), MTAP (2012)]
Recommend news stories based on user’s long-term interests
Example: Video Browsing
[MMM’12-14, Int. Journal of Multimedia Information Retrieval (2014)]
Video Browser Showdown Competition
Example: Other Video Search Engines
Example: Image Browser
[IICAI’09]
Ostensive Browsing
Example: Visual Lifelog Browser
[published as book chapter, Springer Verlag (2013)]
Providing easy access to visual lifelogs
Example: Enterprise Search Engine
[AAMAS’14]
Aggregated Search
Example: Health Information Portal
[PervasiveHealth’13, MIS’14, Book chapter (2015)]
Semantic Search Multilingual Search
Example: Knowledge Management System
[ECIR’14, Book Chapter (2015)]
Targeting intrinsic motivation using Gamification design elements
Multimedia Analysis
Video segmentation
Voice activity detection
Quality assessment
of user-generated
video
Affective content analysis
[ECIR’09, ACM Multimedia (2014), CMBI (‘12, ‘13), MMM (2010, 2013, 2014) , ICMR (2014)]
Data Analysis
Activity and Energy expenditure estimation using accelerometer data
!
Overview
Evaluation Campaigns
Overview NEWSREEL
(CLEF) LifeLog
(NCTIR-12)
Introduction
Career Collaborations Research
How do we evaluate information access systems?
Document
collection
Topic
set
Relevance
assessments
Test co
llection
Document
collection
Evaluation Campaigns
TREC CLEF
FIRE
NTCIR
Common dataset Pre-defined tasks Ground truth Evaluation protocol Evaluation metrics
Focus on different domains
Microblogging
Ad-hoc and Web Search
Multimedia
Federated Web Search
XML Retrieval
Information Access in the Legal Domain
Document Similarity
…
Overview
Evaluation Campaigns
Overview NEWSREEL
(CLEF) LifeLog
(NTCIR-12)
Introduction
Career Collaborations Research
CLEF Tracks
Source: http://www.clef-initiative.eu/
eHealth
ImageCLEF
LifeCLEF
Living Labs for IR (LL4IR)
News Recommendation Evaluation Lab (NEWREEL)
Uncovering Plagiarism, Authorship and Social Software Misuse (PAN)
Question Answering (QA)
Social Book Search (SBS)
CL
EF
’15
In CLEF NEWSREEL, participants can develop news recommendation
algorithms and have them tested by millions of users over the period of a few
months in a living lab.
NEWSREEL
Recommender Systems help users to find items that they were not
searching for.
What are recommender systems?
Items?
Example: YouTube
Example: Netflix
Example: News Articles
Source (Image): T. Brodt of plista.com
What are living labs?
Rely on feedback from real users to develop convincing demonstrators that showcase potentials of an idea or a product.
Real-life test and experimentation environment to fill the pre-commercial gap between fundamental research and innovation.
In an IR context…
“A living laboratory on the Web that brings researchers and searchers together is needed to facilitate ISSS (Information-Seeking Support System) evaluation.”
Kelly et al., 2009
A / B testing
Eval
uat
e
submit to SIGIR
CLEF NEWSREEL
20
14
Frank Hopfgartner
Andreas Lommatzsch
Benjamin Kille
Torben Brodt
Tobias Heintz
Co-Organisers
2015
Frank Hopfgartner
Torben Brodt
Benjamin Kille
Jonas Seiler
Balázs Hidasi
Andreas Lommatzsch
Roberto Turrin
Martha Larson
Scenario
Who are the users?
Devices
• Given a dataset, predict news articles a user will click on
Offline Evaluation
• Recommend articles in real-time over several months
Online Evaluation
CLEF NEWSREEL 2014
TA
SK
1
TA
SK
2
@clefnewsreel http://www.clef-newsreel.org/
Predict interactions based on an OFFLINE dataset
Task 1: Offline Evaluation
DA
TASE
T
EVA
LUA
TIO
N
Traffic and content updates of 9 German-language news content provider websites
Traffic: Reading article, clicking on recommendations
Updates: adding and updating news articles
Recorded in June 2013
65 GB, 84 Million records
[Kille et al., 2013]
Dataset split into different time segments
Participants have to predict interactions of these segments
Quality measured by the ratio of successful predictions by the total number of predictions
Recommend news articles in REAL-TIME
Task 2: Online Evaluation
LIV
ING
LA
B
EVA
LUA
TIO
N
Provide recommendations for visitors of the news portals of plista’s customers
Ten portals (local news, sports, business, technology)
Communication via Open Recommender Platform (ORP)
Provide recommendations within <10ms (VM provided if necessary)
Three pre-defined evaluation periods
5-23 February 2014
1-14 April 2014
5-19 May 2014
Evaluation criteria
Number of clicks
Number of requests
Click-through rate
Living Lab Scenario
…
Publisher A
Publisher n
Researcher 1
Researcher n
…
plista ORP
…
Millions of visitors Publishers Teams
Open Recommender Platform
Number of clicks
[CLEF’14]
Number of requests
[CLEF’14]
Click-Through Rate
[CLEF’14]
Overall results
Advertisement: Join the Living Lab - Tutorial at ECIR’15
Overview
Evaluation Campaigns
Overview NEWSREEL
(CLEF) LifeLog
(NTCIR-12)
Introduction
Career Collaborations Research
NTCIR
Sourc
e: H
ideo J
oho
NTCIR-12 Tasks
NT
CIR
-12
Second round:
Search-Intent Mining
Mobile Click
Temporal Information Access
Spoken Query & Spoken Document Retrieval
QA Lab for Entrance Exam
First round:
Medical NLP for Clinical Documents
Personal Lifelog Access & Retrieval
Short Text Conversation
Encourage research advances in organising and retrieving from lifelog data.
LifeLog @ NTCIR-12
Lifelogging Challenges
The challenges are how to sense the person, their actions, their life and make it accessible using appropriate interfaces, search, recommendation engines and visual/aural feedback. Further, exploiting the lifelog to identify context for adaptive information services.
Source (Graphic): DAI-Labor, Berlin
LifeLog @ NTCIR-12
CO
-OR
GA
NIS
ER
S
TIM
EL
INE
Cathal Gurrin, Dublin City University
Hideo Joho, University of Tsukuba
Frank Hopfgartner, University of Glasgow
Brian Moynagh, Dublin City University
Rami Albatal, Dublin City University
27 Feb 2015: Kickoff event
30 Jun 2015: Task Registration
01 Jul 2015: Dataset release
Sep 2015 – Feb 2016: Formal run
01 Feb 2016: Evaluation Results
01 Mar 2016: Paper for the Proceedings
7-10 Jun 2016: NCTIR-12 conference
Multimodal dataset with information needs
Created by 5-10 individuals over
10+ days
TE
ST
CO
LL
EC
TIO
N
1,500 images, location, GSR, heart-rate, others… per lifelogger per day
Accompanying output of 1,000 concepts
Data processed pre-release (removal of personal content; face blurring, translation of concepts)
Detailed user queries and judgments generated by the lifelogging data gatherers
Evaluate different methods of
retrieval and access.
Tasks
T1:
LIFE
LOG
SEM
AN
TIC
AC
CES
S (L
SAT)
T2:
LIFE
LOG
IN
SIG
HT
Models the retrieval need from lifelogs (Known-Item Search)
Retrieve N segments that match information need
Interactive or Automatic participation
Interactive: Time limit for fair and comparative evaluation in an interactive system with users
Automatic: Fully-automatic retrieval system. Automated query processing
Models the need for reflection over lifelog data
Exploratory task, the aim is to:
encourage broad participation
novel methods to visualise and explore lifelogs
Same data as LSAT task
Presented via demo/poster.
Thank you
Come and talk to me
Frank Hopfgartner, PhD [email protected]
@OkapiBM25 www.hopfgartner.co.uk