Upload
caren-hicks
View
224
Download
4
Embed Size (px)
Citation preview
Identifying Programmer Ability Using Peer Evaluation: An Exploratory Study
Jeffrey C. Carver, University of AlabamaLorin Hochstein, USC Information Sciences Institute
Jason Oslin, University of Alabama
The Importance of Categorizing Programmer
Ability
• In Industry
• In Research
• In Teaching
2
How We Measure Ability
• Professional vs. Novice
• Years of Experience
• Elaborate Evaluations
3
Is There A Better Way?
• Peer evaluation
• “We know a good programmer when we see one”
• Subjective judgment as a predictor
4
Outline
• Study Design
• Data Analysis
• Conclusions
• Future Work
5
Our Study
• Goal– Characterize peer assessment of
programmer ability from the point of view of a researcher in the context of an undergraduate computer science course
• Hypotheses–H1: Developers are able to accurately
predict the programming ability of their peers
–H2: Low-performers tend to over-estimate their ability
6
Test Environment
Senior Level Programming Languages Course
– Five individual assignments
– Graded by the unbiased instructor
– Student participation had no effect on grade
7
Data Collected
• Subjective Data– Participant evaluation of self– Participant evaluation of peers– Post-study questionnaire
• Objective Data– Grades on projects– “True” measure for this study
8
Collecting Subjective Data
9
Sample data collection form
Coding the Grades
10
• A = 5 Top 20%• B = 4• C = 3• D = 2• E = 1 Bottom 20%
Coding the Grades
11
A Look at the Data Set
12
Raters
Ratees
A Look at the Data Set
13
Evaluating Hypothesis 1Developers are able to accurately predict
the programming ability of their peers
14
Act
ual
Gra
de
Mode Of Peer Ratings
Additional Observation
15
Actual Grade
Diff
ere
nce
Betw
een
Mod
e a
nd
Act
ual
Evaluating Hypothesis 2Low-performers tend to over-estimate their
ability
16
Actual Grade
Self Rating
A Qualitative Analysis
• Rating oneself
• Compare my self perception to my peers’ perception of my ability
• Confidence in rating peers
17
Threats to Validity
• Construct– Are grades a valid indicator of ‘true’
ability?– Ceiling effect displayed by skew of grades?– Are we evaluating enough skill sets
involved in software development?
• Low number of ratings• What happens when we move beyond
from classrooms to industry?
18
Future and Ongoing Work
• Running the study again
• Goals of future work– Characterize high-performers–Understand programming ability– Improve competent programmers
19