View
214
Download
0
Category
Tags:
Preview:
Citation preview
Contextual Wisdom
Social Relations and Correlations for Multimedia Event Annotation
Amit Zunjarwad, Hari Sundaram and Lexing Xie
Talk Outline
Observations
Events
Generalization:Sum of Partial Observations
Similarity, Co-Occurrence and Trust
@I2R April 18, 2023 3
Experiments:compare against SVM
Conclusions
The pool statistics reveal a power law distribution• Less than 11% of the tags have more than 10 photos• There are not enough instances to learn most of the
concepts! The global flickr pool is interesting:
April 18, 2023 6@NUS
Learnability
The pool statistics reveal a power law distribution• Less than 11% of the photos have more than 10
instances• There are not enough instances to learn most of the
concepts! The global flickr pool is interesting: • Most of the tags have over 100 instances• Photos reveal very high visual diversity
The Power law is a fundamental property of online networks – cannot be wished away.
April 18, 2023 8@NUS
Learnability
Singapore People Walking Orchard rd. After MRT Experimenting Walking Day Outdoor..
April 18, 2023 9@NUS
Scalability
The assumption of consensual semantics Search for “yamagata”
April 18, 2023 10@NUS
The Role of context
An event refers to a real-world occurrence, spread over space and time.
Observations form event meta data [Westermann / Jain 2007]• Images / text / sounds describe events
April 18, 2023 13@NUS
Defining Events
when
where
who
what
author
image
Event context refers to the set of attributes that help in understanding the semantics • Images / Who / Where / When / What / Why / How
Context is always application dependent • Ubiquitous computing community – location, identity
and time are main considerations
April 18, 2023 14@NUS
Context
[Mani and Sundaram 2007]
Event archival – events involve people, places and artifacts
Exploit different forms of knowledge: • (Global) Similarity – media, events, people. • (Personal) Co-occurrence – what are the joint statistics
of occurrence?• (Social) Trust – determining whom to trust for effective
annotation?
April 18, 2023 15@NUS
Four Problems
A bottom up approach • Edge, color and texture histograms for images. • Rely on ConceptNet for text tags
Why ConceptNet and not WordNet?• Expands on pure lexical terms, to compound terms –
“buy food”• Expands on number of relations – from three to twenty• Contains practical knowledge – we can infer that a
student is near a library.
April 18, 2023 17@NUS
Event similarity
ConceptNet provides three functions:• GetContext(node): the neighborhood of the concept “book” includes “knowledge”, “library”
• GetAnalogousConcepts(node): concepts that share incoming relations; analogous concepts for the concept “people” are “human”, “person”, “man”
• FindPathsBetweenNodes(node1,node2) – returns a set of paths.
Our similarity measure is built using these functions.
April 18, 2023 18@NUS
A base similarity measure
The similarity between two concepts (e,f) is defined as follows:
We current use a uniform weighting on all three as the composite measure
April 18, 2023 19@NUS
Concept similarity
fe
fCeC
eA fA
context
analogous
path based1
1 1( , )
N
pi i
s e fN h
The distance between two concept sets is a modified Haussdorf similarity.
April 18, 2023 20@NUS
Computing similarity between sets
A
B
| |
1
1( , | ) max{ ( , )}
| |
A
H k ii
k
S A B m m a bA
Similarity between facets are computed using a weighted sum of frequency and the concept similarity measure:
Time distance is based on text tags, not actual time data – allows for temporal descriptions as “summer”, “holidays” etc.
Only frequency is used for “who” facet.April 18, 2023 21@NUS
Facet similarity (4w)
1 21 2 1 2
2
1 | |( , ) , | ,
2 | | H
L Ls L L S L L CS
L
Color, texture and edges are computed• 166 bin HSV color histogram• 71 bin edge histogram• 3 texture features
Euclidean distance on the composite feature vector.
The distance between two events is then a weighted sum of distances across all event facets.
April 18, 2023 22@NUS
Image facet similarity
The concept co-occurrences are just frequency counts.
(i= fun , j = new york) then the index (i,j) contains the number of occurrences of this tuple.
Notes:• Each concept is given a globally unique index• Co-occurrence matrixes are locally compact
Each user k, has a co-occurrence matrix Mck
associated with the user.
April 18, 2023 25@NUS
Statistics are computed per person
Narrow understanding of “trust” a priori value is important Computing trust:• Compute event-event similarity
Trust propagation• Biased PageRank algorithm
• Trust vectors are row normalized
April 18, 2023 27@NUS
Activity based trust
activity matrix apriori
(1 ) k t = A t + p
The framework is event centric We know:
How to combine the three?
April 18, 2023 29@NUS
A review of what we know
similarity co-occurrence trust vectors
global personal social
1. Compute the social network trust vector (t) for the current user.
2. Compute the trusted, global co-occurrence matrix, for all tuples.
3. Iterate:
April 18, 2023 30@NUS
details
1
( , ) ( ) ( , ),N
k kc c
i
a b t i a b
M M
who where whatwhen image event
query
,
,c
s
y M x q
x M y q
Developed and event based archival system
8 graduate students 58 events, 250 images,
over two weeks SVM – baseline
comparison Two cases• Uniform trust (global)• Personal trust
April 18, 2023 32@NUS
Details
Training is difficult – very small pool.• Modified bagging strategy • Train five symmetric classifiers• Pick one which maximizes the F-score
April 18, 2023 33@NUS
SVM training
Global Case:• 31 classifiers (who:8, when: 6, where: 10, what: 7)• Minimum number of images: 10 • Tested on 50 images (why?)
April 18, 2023 34@NUS
Uniform trust
Facets SVM CM (uniform)
H M X U H M
Who 13 23 5 9 22 28
When 11 20 6 13 24 26
Where 12 19 3 16 23 27
What 13 21 8 8 31 19
Event 10 12 22 6 22 28
H Hits
M Misses
X Unknown
U Undecidable
Trained classifiers per person• Very small pool• Min images – 5• 28 classifiers (who:9, when: 4, where: 6, what: 9)
April 18, 2023 35@NUS
Personal Network
Facets SVM CM (network)
H M X U H M
Who 45 81 62 62 183 67
When 51 96 73 30 167 83
Where 62 76 59 53 179 71
What 72 89 23 66 204 46
Events 0 0 250 0 153 97
H Hits
M Misses
X Unknown
U Undecidable
An event based annotation system• Media are event meta-data• Issues: learnability, scalability, context
Employ three kinds of knowledge• Global – conceptnet, image similarity• Personal – statistical co-occurrence • Social – trust
Recommendations• Employ iterative schemes (HITS / PageRank)
Results:• Outperform SVM in small pools
April 18, 2023 42@NUS
summary
Power law tag distribution• Data pool will remain small for most tags• Fundamental issue
Participatory knowledge is powerful – trust within context is important issue.
Future work: • Careful math analysis of coupling equations• Event structure / relationships need to be incorporated • Multi-source (email / Calendar / IM / blogs)
integration.
April 18, 2023 43@NUS
Conclusions
Recommended