Does Studio-Based Instruction Work in CS 1 - An Empirical Comparison With a Traditional Approach

Does Studio-Based Instruction Work in CS 1? An Empirical Comparison with a

Traditional Approach*

*This work is funded by a National Science Foundation CPATH Award (CNS-0721927)

Christopher Hundhausen* Anukrati Agrawal*, Dana Fairbrother, and Michael Trevisan

*School of Electrical Engineering and Computer Science College of Education

Washington State University Pullman, WA 99164

[email protected], [email protected], [email protected], [email protected]

SBL

Traditional Approaches Dont Address Skills Needed by Computing Professionals

CS 1 traditionally emphasizes individual programming assignments

New jobs in computing require design, communication, teamwork, collaboration, and critical thinking skills (see also Carl Wieman lecture)

2 Introduction Related Work Quasi-Experimental Study Summary & Future Work

Studio-Based Learning (SBL) Approaches Can Address Mismatch Adapted from

architectural education Actively engage

students in four essential activities Solution construction

(problems should have multiple solution paths)

Solution presentation Peer Review Peer Collaboration


Student Instructor

Peers

Pedagogical Code Review Is One Possible SBL Approach

Based on formal code inspection (Fagan, 1986)

Team of students first review each others code solutions individually

They then come together to identify, discuss, and log issues with the code (defects and improvements)

Students may resubmit their code solutions based on the reviews


Do Pedagogical Code Reviews Work?

Previous study (Hundhausen et al., 2009) furnished preliminary evidence that PCRs Improve students code solutions Engage students in educationally beneficial

discussions Promote a sense of community

A more rigorous empirical evaluation of educational impact is needed!

5

6

Remainder of Talk

Background and related work Quasi-experimental study Discussion Summary and Future Work

Introduction Related Work Quasi-Experimental Study Summary & Future Work

Studio-based Approaches Have Been Integrated into Science and CS Ed.

Science instruction at high school (Faro & Swan, 2006) and undergraduate level (Lister, 2001)

Individual computing courses at the undergraduate level (Myneni et al., 2008, 2010; Reimer & Douglas, 2003)

Entire computing degree programs in Australia (Lynch et al., 2000; Docherty et al., 2003)


Our Pedagogical Code Review (PCR) a Form of Peer Review

Peer reviews have been explored by many computing educators, e.g., Pair programming (Williams & Kessler, 2001) In-class conference review (Kern et al., 2003) Peer-reviewed group projects (Anewalt, 2005) Peer review in CS 1 course labs (Trytten, 2005)

Several web-based systems have been developed to support peer review RRAS (Trevidi, 2003) PeerGrader, Expertiza (Gehringer, 2001, 2006) Reily et al. (2009) system (solid empirical study!)


Quasi-Experimental Study Tested Two Hypotheses

H1: Students who participate in PCRs will learn computer programming significantly better than students who do not

H2: Students who participate in PCRs will experience positive shifts in their attitudes toward learning that are significantly higher than those who do not


Quasi-Experimental Study Defined Two Treatments

Traditional (Fall, 2008 offering of CS 1) Studio (Spring, 2009 offering of CS 1) Treatments defined by presence or absence

of three pedagogical code reviews (PCRs) Treatments identical in nearly every other

respect Same instructor, textbook, lecture slides,

assignments, labs Slightly different quizzes and tests (to discourage

cheating)


We Collected Data on Two Dependent Measures

Learning Outcomes Pre-/post test of course content

20 multiple-choice questions (tracing code and identifying elements of code)

10 short-answer (writing and tracing code) Attitudes

Motivated Strategies for Learning Questionnaire (Pintrich et al., 1991)

Sense of Community Questionnaire (McKinney et al., 2006)

Exit Interviews Exit Questionnaires (Studio treatment only)


Study Participants

Traditional treatment: 87 students Studio treatment: 89 students But, because we required informed consent,

we collected data on smaller numbers of students:


Study Procedure

Week 1: Pre-survey and pre-test Weeks 8, 11, and 13: Pedagogical code

reviews (Studio treatment) Week 8: Number processing (~150 lines of code) Week 11: Battleship (~500+ lines of code) Week 13: String library (~250 lines of code)

Week 15: Post-survey and exit interviews (2 low, 2 mid, 2 high)

Final exam: Post-test (embedded in final exam)


PCRs Were Modeled After Formal Code Inspections

Teams of 3-4 students led by moderator

Code reviewed against a list of best practices + list of requirements of specific assignment

Team members rotated through 4 roles: reader, inspector, recorder, author


Results: Learning Outcomes

Both treatments made significant gains; however, gains did not differ significantly

Note: Scores were out of 60


Results: Attitude Survey

16

(df = 1, F = 5.68, p = 0.0262) (df = 1, F = 0.72, p = 0.4065)

(df = 1, F = 3.78, p = 0.059)


Results: Exit Interviews

Helpful feedback received? All four Studio respondents said yes Five of eight Traditional respondents said yes

Sense of shared experience? All Studio respondents said yes Half of Traditional respondents said yes

Course experience made them feel comfortable interacting with others about programming? All Studio respondents said yes Five of eight of Traditional respondents said yes


Results: PCR Exit Questionnaires

85 percent found PCRs of their own code helpful It was much easier to identify errors in a group Others might see an error I overlooked I learned to appreciate criticism

87 percent found PCRs of others code helpful gave me ideas on how to go about this program next time I learned it will help to work with people

96 percent said their team worked well together and they would work with a team member in future [My team] understood where each other is coming from and

didn't take offense to the critique of each others programs. My voice was heard. I was able to listen to and appreciate

others opinions.


19

Discussion: Evidence that PCRs Positively Influenced Attitudes

Why did self efficacy diminish in Traditional treatment but not in Studio treatment? Bandura (1997): enactive experiences enable

learners to assess their proficiency PCRs were powerful enactive experiences

Why did peer learning increase in Studio treatment at level approaching significance? PCRs are form of peer learning Studio treatment had more opportunities for peer

learning Students had positive impression of those

opportunities Introduction Related Work Quasi-Experimental Study Summary & Future Work

20

Summary

PCRs are a form of studio-based instruction, which has a rich heritage of use in architecture, science, and computing education

Little research has rigorously evaluated educational impact

We have presented a quasi-experimental study with mixed results


21

Future Work

Better understand impact of PCRs through analysis of video data (in progress)

Develop better ways of assessing the kinds of teamwork, communication, and critical review skills that we believe to be promoted by studio-based instruction

Rigorous evaluation of studio-based instruction in broader range of courses, using a broader range of SBL approaches (in progress, funded by NSF CPATH Class II)


22

Interested in Learning More About Studio-Based Learning?

We are holding an information session on studio-based learning and our research project at the conclusion of this symposium 3 6 pm in Lakeshore Meeting Rooms in Hyatt

Join our community! Please talk to me after the session if interested Visit the SBL portal and sign up for a login at

http://www.studiobasedlearning.org

Studio-Based Instruction Has Been Around For a Long Time

Roots: Master-apprentice educational system used in guilds of middle ages

Architecture and fine arts educators adapted this model in form of design studio Camaraderie Collaboration Design crits (critiques)


Documents

Does Studio-Based Instruction Work in CS 1 - An Empirical Comparison With a Traditional Approach