Upload
others
View
9
Download
0
Embed Size (px)
Citation preview
EXYSTENCE Topical WorkshopTrust-Based Networks and Robustness in Organizations
ETH Zurich, March 15, 2006
Collective Evaluation: Practice and Theory
Yi-Cheng Zhang (Fribourg University, Switzerland)
Collective Evaluation: Practice and Theory
Main Themes●New generation of Web Sites● From— People—For—People
Wikipedia(user generated encyclopedia)de l.icio.us (social bookmarks)
Ohmynews.com society digg.com reddit.com technocrati.com
flickr.comblogdex.com
SWITZERLAND
Collective Evaluation: Practice and Theory
GoogleAmazon.comEpinions.comRatemyteachers.com
...professor.comEconomy ...doctor.com Ebay.com
ZAGAT Guides(restaurant) Dianping.com
Douban.com
.
SWITZERLAND
Collective Evaluation: Practice and Theory
Successful Case impacting all of us:Google, Amazon.com, Ebay.com,
Wikipedia.org, Slashdot.org, World66.com....
(new “animals ” are added fast by the day,explicitly&implicitly using “New” info-Theory)
Google: (Page Rank) by S. Brin & L. Page and by Kleinberg( 96~98)
Hubs vs Authorities
SWITZERLAND
Collective Evaluation: Practice and Theory
Old Model:newspapers, government, agency.
New Model:“Matchmaker site”
SWITZERLAND
Collective Evaluation: Practice and Theory
Objects can be:
books, products, news, stories, tourist places, ideas, favorite actress/actors, music, scientists, teachers(!)
SWITZERLAND
Collective Evaluation: Practice and Theory Advantages Disadvntages Old expertise corruption
easy control manipulation
New people are not noisy corrupted!!! contradictory distortion everyone is good at pseudo-experts something, sometimes laziness even better than experts Not reliable!!!
SWITZERLAND
Collective Evaluation: Practice and Theory
Science may help?Objective: to get the best approximation possible to assure quality information.
1) pure democracy one-man-one-vote2) authority; people NO influence
our challenge: find the best balance of the two- distribution of weights
SWITZERLAND
Collective Evaluation: Practice and Theory
Simple Model (XY):
SWITZERLAND
Q l l=1 ,2 , .... , M Intrinsic quality ( “God-Given)”
ql Approximate evaluation
X il i=1 ,2 , ... , Nl=1 ,2 , ... , M
“i”th expert's evaluationon object α
Y i “God -Given”judging capacity
Naive democratic method:
“Tyranny of Majority”!
More refined approach:
Collective Evaluation: Practice and Theory
SWITZERLAND
ql=1N∑i=1
N
xil
ql=∑i=1
N
w i x il w i : Weight of “i”th expert
What weight to assign?
Collective Evaluation: Practice and Theory
ql=∑i=1
N
w i x il
i2≈V i≡
1M∑
l=1
M
xil−ql2
SWITZERLAND
Collective Evaluation: Practice and Theory
∑i=1
N 1V i
x il−ql =0 l=1 ,2 , ...M ;
∑l=1
M x il−q l2
V i=M i=1 ,2 , ...N
SWITZERLAND
Rewriting as:
Collective Evaluation: Practice and Theory
SWITZERLAND
= 1M∑
l=1
M
ql−Q l 2
I p≡ 1[ pM ] ∑R=1
[ pM ]
nR/R
[ I p]0=[ pM ]1
2M p /2 M≫1
Collective Evaluation: Practice and Theory
“New” Information Theory for the Internet Age
Classical Information Theory (Claude Shannon 1948)
Original message
SWITZERLAND
0 0 0 1 0 1 0 0 0 1 1 0 0 1
noise transmission
1 0 0 1 1 1 0 0 0 1 1 0 0
Error
Need of redundancy(large number repeats)
Collective Evaluation: Practice and Theory
The goal:
SWITZERLAND
Use the least number of repeats to recover original message
Give a finite number of repeats, get the best approximation
Collective Evaluation: Practice and Theory
Challenge of Noise
Classical Information Theory: natural noise
“New” Information Theory:natural noise
intentional noise herding noise
SWITZERLAND
Collective Evaluation: Practice and Theory
When objects are too complex, need intermediate layers XY→XYZ
Amazon.com example
book 1 book 2.... ...... ....... ......book M
reviewson books
Readers
Reviews are easier than books to evaluateSWITZERLAND
Collective Evaluation: Practice and Theory
Collaborative Filtering & Predictions
Challenge: “Movielens Project”
(Michigan University 1997-2000)
3 Millions data 2000 movies rated by 17'000 viewers
SWITZERLAND
Collective Evaluation: Practice and Theory
example
Movies
123456
8
109
1 2 3 4 5 6 .. .. .. .. .. ... .. 20
√
√
√
√
√√
√
√
√
√√
√7
Viewers
√√
√
√
√
√
√
√
SWITZERLAND
Collective Evaluation: Practice and Theory
SWITZERLAND
v i Viewer i votes on movie α
I, j = 1,2,...Nα, β =1,2,...M
v i=1 ,2 ,3 , 4 ,5 Stars
Collective Evaluation: Practice and Theory
SWITZERLAND
ij=∑=1
M
vi v j
∑=1
M
∣vi∣∣v j ∣, ij∈−1 ,1
Person-to-person relationship:
d ij=1−ij/2Distance
Collective Evaluation: Practice and Theory
=∑i=1
N
vi vi
∑i=1
N
∣vi∣∣vi ∣, ∈−1 ,1
d =1−/2
Movie-to-movie relationship:
Distance
SWITZERLAND
Collective Evaluation: Practice and Theory
SWITZERLAND
2 23...
'=
1−
More Refined Approach
Collective Evaluation: Practice and Theory
SWITZERLAND
=∑ij∑S i
∣V test∣
≃0.83 0.9
Predictions
Collective Evaluation: Practice and Theory
Theory Group for“Interdisciplinary Physics”
Post.doc. Paolo Laureti (Roma) Ph.D. Marcel Blattner (Zürich)
Ph.D. Lionel Moret (EPFL)Post.doc. Hasan Masum (Ottava, Canada)
SWITZERLAND
Fribourg Members:
Collective Evaluation: Practice and Theory
SWITZERLAND
External Members:
Yi-kuo Yu (NIH, Washington)Damien Challet (Oxford)Matteo Marsili (ICTP; Italy)Sergei Maslov (BNL, New York)
Collective Evaluation: Practice and Theory
Thank You very much for your attention!
SWITZERLAND
Workshop: Internet @nd Information Filtering Mechanism
SWITZERLAND
Workshop: Internet @nd Information Filtering Mecanisme
SWITZERLAND
Workshop: Internet @nd Information Filtering Mechanism
much harder... but our task is modest:
best approximation,can not dream about decoding “God-Given”
SWITZERLAND