Peer Assessment Based on Ratings in a Social

Media CourseAndrii Vozniuk, Adrian Holzer, Denis Gillet

Andrii Vozniuk, Adrian Holzer, Denis Gillet. “Peer Assessment Based on Ratings in a Social Media Course”. In the proceedings of LAK’14.

The copyright of images belong to their authors. I will remove them on demand. Contact me at adrian.holzer@epfl.ch

SpeakUp

www.seance.ch

Is peer assessment reliable ?

Practice what you preachsocial evaluation

in a social media course !

using a social media app

Social Media Course

age 23-2860 students

Mining the social web

Assignment

Goal

Extract relevant information for the class from a social network in the context of the Long Tail and plot the result

#fol

low

ers

Ranked users

Individual work2 pages

Examples

Grading

Your grade is strongly influenced by the grade from your peers, but might differ

5 - I love it, it’s amazing4 - I like the report3 - It’s OK2 - I don’t like it1 - I hate the report

as in social media

Criteria1 - Is the report about a social media?

3 - Is the dataset representative?4 - Is it technically advanced?5 - Is the report well written?

2 - Is the long tail hypothesis clear?

6 - Is it interesting and creative?

Graaspcreate a space,

drop resources & apps, invite people to collaborate

Reviewers

60reports each

3 instructors

20 reports each

60 students

40 kids

15 reports each

11 years old

StatisticsConsensus Group meanAgreement Cohen’s weighted Kappa

Results

Instructors vs Kids

Kids vs Students

Instructors vs Students

AgreementWeighted

KappaLower Bound

(95%)Upper Bound

(95%) Sig Level

0.196 -0.59 0.427 0.065

AgreementWeighted

KappaLower Bound

(95%)Upper Bound

(95%) Sig Level

0.285 0.035 0.501 0.013

AgreementWeighted

KappaLower Bound

(95%)Upper Bound

(95%) Sig Level

0.774 0.648 0.858 0.00001

ConclusionPeer assessment seems quite reliable

Based not only on appearance

Working on the full integration in Graasp

Thanks!