Feedback and Free(?) Snacks - Computer ScienceJul 19, 2012 · Feedback and Free(?) Snacks. Björn Hartmann University of California, Berkeley EECS, Computer Science Division July

Björn HartmannUniversity of California, Berkeley

EECS, Computer Science DivisionJuly 2012

Two Experiments in Crowd Education

Feedback and Free(?) Snacks

mailto:[email protected]


Björn HartmannUniversity of California, Berkeley

EECS, Computer Science DivisionJuly 2012

One Experiment in using Crowds for Education

Free (?) Snacks



UW MSR Summer Institute 2012 3

Crowds & Collective Intelligence

Expertise Sharing & (Informal)Learning

Crowd Platforms(Turkomatic, MobileWorks)

Communitysourcing(Local Crowds Grading Exams)

Expertise Sharing for End-User Programmers(HelpMeOut,

HyperSource)

Improved Design Tutorials

(ShowMeHow, Delta, Sifter, MixT)

Crowd Applications(BribeCaster, VeriFun)

Empirical Analyses

(Stack Overflow)

Teaching Crowd Workers(Shepherd)






HyperSource)




Empirical Analyses

(Stack Overflow)









HyperSource)




Empirical Analyses

(Stack Overflow)



Word 2011

Word 2008

UW MSR Summer Institute 2012

Similar App, Different Command

?10


ShowMeHow Interface Model Interface Model

Translation Map

Interaction Techniques

11






HyperSource)




Empirical Analyses

(Stack Overflow)


expert

creative, multi-faceted

skill level

work complexity

novice

simple, transactional

Shepherd: Training Crowd Workers

w/ Steven Dow, Anand Kulkarni, Scott Klemmer

expert

creative, multi-faceted

skill levelnovice

simple, transactional

work complexity


Worker


Task submission

Assessor

Shepherd feedback

Assessment helped workers improve over time

95% HPD interval = [0.09, 0.44], p=0.001

0

0.1

0.2

0.3

0.4

0.250.10Slop

e Co

effici

ent

of R

atin

gs O

ver T

ime

external assessment

self assessment

no assessment

95% HPD interval = [-0.03, 0.23], p=0.08

95% HPD interval = [-0.13, 0.06], p=0.38

External assessment encourages more revisions

0

20

40

60

80

100

24.856.5

External Self

Percent of reviews edited

χ2=47.3, p<0.05

0

32

64

96

128

160

84137

Length change (chars)

F(1,81)=3.26, p=0.07

External Self

%






HyperSource)




Empirical Analyses

(Stack Overflow)



CommunitySourcing

19

Engaging Local Crowds to Perform Expert Work via Physical Kiosks

with Kurtis Heimerl, Brian Gawalt, Kuang Chen, Tapan Parikh


Goal

Crowdsource the grading of (non-autogradable) examsRequires expert knowledgeVery time intensive - high volume

20


Microtask Markets

Examples: Mechanical Turk, Crowdflower…Recruit from large pool of unskilled workersSafety in numbers: issue multiple assignments,aggregate the responses

2 2

2

1

2 3= 21+1 = ?

21


Microtask Market Benefits

Pros:High Task VolumeQuick ResponsesSupport for Small Tasks

Cons…

22


Microtask Market Drawbacks

Lack of expert workers(experts have full-time jobs)Hard ceiling on knowledge tasks?

4 22

x

-1

9 -2 = ?

23

eπ=


Freelancing

Examples: Elance,oDesk…Hire the best applicant (as judged by you)

1 yr 0

2 mo

3 mo

3 yr 6 mo

How muchgraphic design experience do you have?

24


Innovation Contests

Examples: 99Designs, InnoCentive, TopCoder…Ask a set of experts a questionTake the best response (as judged by you)

a o

q

c

d p

Design a new logo.

25


Innovation Contests & Freelancing

Pros:Expert, high quality workSupports large, complex tasks

Cons:Short tasks don’t workLow volume

26


How might we crowdsource short, complex work at high volume?

27


CommunitySourcing

1 Bring the crowdsourcing to the crowdPlace a work kiosk where communities of these experts congregate and have free time

28


CommunitySourcing

1 Bring the crowdsourcing to the crowd2 Pay them in what they valueProvide rewards that fit the users and their context, not just small change

29


CommunitySourcing

1 Bring the crowdsourcing to the crowd2 Pay them in what they value3 Not one big market - many small onesTarget workers with tasks that are situated in the context and location

30


Task: Grade CS Exams

Community: Computer Science students

31



Community: Computer Science studentsLocation:Outside Main CS lecture hall

32



Community: Computer Science studentsLocation:Outside Main CS lecture hallReward: (Unhealthy) Food.

33


Touchscreen

Cardreader

Micro-controller+ Laptop






Exams

Generated a 13 question exam out of prior CS2 exams available online8 Participants + 1 “Ringer”

39


Exams

105 questions answered in totalAnswers were scanned and chunked by captricity.comEach question graded on a 0-4 point scale

40


Example Questions

Q: Briefly explain the difference between a (Java) instance variable and a (Java) class variable.

A: Each object has its own distinct copy of an instance variable. But all the objects in a class share just one 'copy' of a class variable.

41


Answer Key

GradingSlider

Question

StudentAnswer



Rewards (in Umati credits)

44

10 – Small granola bar

20 – Normal candy

30 – Energy bar


1 Week300+ Unique Users7500+ Answers Graded

$200 of Candy


Comparisons

Mechanical TurkRebuilt interface using web

46


Comparisons

Expert graders (n=10), from our university

47


Comparison: Mechanism

Gold standard: median grade of all experts

We sampled (with replacement) from each distribution of raters (experts, umatiers, turks) and compared to the gold standard for each exam question.

48


Accuracy

49


Accuracy

50

O1 Expert: 78.3%


Accuracy

51

O3 Umatiers: 71.2%


Accuracy

52

O 5 Turkers:25.6%


Accuracy

53

O O 10 Umatiers:78.9%

1 Expert: 78.3%


Results - Cost

54

Umati: 2.25¢/answer; Turk: 2.3¢/answer; Expert: 34¢/answer


Results – Lower Cost

55

OO

11¢/grade

{


Results – Higher Accuracy

56

O

O2%

{


Potential Biases

Novelty effect likely330 users in 4 days!However, usage was growing at study end…

Computer Science Bias: Students more likely to crowdsource?

57


Conclusion

CommunitySourcing works!Accuracy++/Cost-- if you…Place a crowdsourcing kiosk within an expert communityProviding rewards relevant to that communityTargeting expert skills available in that community

58


Open Questions

Scaling: How do we support concurrent usage without losing “community” feel?Location: How important is it that the tasks are local (vs. just the rewards?)Generalizability: What other tasks/communities/rewards fit with communitysourcing?

59

[email protected]

http://hci.berkeley.edu

http://hci.berkeley.edu

Documents

Feedback and Free(?) Snacks - Computer ScienceJul 19, 2012 · Feedback and Free(?) Snacks. Björn Hartmann University of California, Berkeley EECS, Computer Science Division July