Johnny can’t Program (and neither can Johann) Lynda A. Thomas Department of Computer Science, University of Wales, Aberystwyth, Wales, UK

Johnny can’t Program(and neither can Johann)

Lynda A. Thomas

Department of Computer Science, University of Wales, Aberystwyth, Wales, UK

Slide 2 of about 50

Two Multi-institutional and Multi-national Studies

of Beginning Programmers(and some attempts at solutions)

• A multi-national, multi-institutional study of assessment of programming skills of first-year CS studentsReport by the ITiCSE 2001 Working Group Mike McCracken (et al.) SIGCSE Bulletin (Dec 2001)

• A multi-national study of reading and tracing skills in novice programmers Report by the ITiCSE 2004 Working Group Raymond Lister (et al.) SIGCSE Bulletin (to appear)

• Various papers by Lynda Thomas and Mark Ratcliffe

Slide 3 of about 50

Outline

• Outline the problem

• Report on McCracken Study

• Report on Lister Study

• Some possible remedies

• Discussion

Slide 4 of about 50

Programming is hard!

• du Boulay (1989) describes the sources of difficulty, and the need to deal with all of them at once: General orientation, notional machine, notation, structures, pragmatics

• Soloway and Spohrer (1989) note deficits in understanding of concepts, shortcomings in planning and testing, and much more

• Winslow (1996) notes that novices lack detailed mental models, are limited to surface knowledge and approach the problem line-by-line as opposed to in chunks

Slide 5 of about 50

Does OO make it worse?

• Rist (1996) suggests that is not so much that OO is different, it is more

• Wiedenbeck (1999) compared OO and Procedural novices and discovered that although for short programs there was little difference in comprehension, programs with multiple classes were not understood as well.

• However, in a later study it appeared that almost all difference was coming from weaker students.

Slide 6 of about 50

On a Personal Note

• 20 years of experience, US and UK• Old assignments - would I give them now? (note

how more comes into this (e.g.. solitaire))• There is a lot more to teach• The population is different

Many of my students cannot program (30%?)

Is it my fault?

Slide 7 of about 50

McCracken Study

• Participants from 8 universities, 5 countries were given a choice of 3 problems that students ‘should’ be able to solve (the problems were calculators).

• Then gave the problem to an entire class of students at the appropriate level.

• Solutions were marked by several researchers.

Slide 8 of about 50

McCracken, et al. (2001)

• Most were end of first year students• Remember, 8 universities, 5 countries …• … it says something about our discipline.

0

5

10

15

20

25

30

35

40

1 8 16 24 32 40 48 56 64 72 80 88 96

Scores

Nu

mb

er o

f S

tud

ents

The “Picassos”“Finger painters”

Slide 9 of about 50

Why???• According to McCracken et al, solving a programming

problem involves the following …

– (1) Abstract the problem from informal description,

– (2) Generate sub-problems,

– (3) Produce sub-solutions,

– (4) Re-compose, and

– (5) Evaluate and iterate

Slide 10 of about 50

Why???• According to McCracken et al, solving a programming problem

involves the following …

– (1) Abstract the problem from informal description,

– (2) Generate sub-problems,

– (3) Produce sub-solutions,

– (4) Re-compose, and

– (5) Evaluate and iterate

“Oh, my students know how to code, but they can’t design.”(or “problem solve”)


Lister Study

• The nature of the McCracken study does not isolate the exact problem

• This study attempted to separate the steps so that we could see whether the problem was at the problem solving/design step or at the programming level

• Students attempted 12 multiple choice questions.– Generic, 3GL, iterative processes on arrays– Could be translated into many languages, as it happens …Java (11

universities) and C++ (1 university)– 5 from US, 3 from Scandinavia, 3 from Australasia , 2 from UK

After the while loop finishes, “count”contains what value? a) 3b) 2c) 1d) 0

(Formatting compressed to fitquestion into this space.)

int[] x1 = {1, 2, 4, 7};int[] x2 = {1, 2, 5, 7};int i1 = x1.length-1;int i2 = x2.length-1;int count = 0;while ((i1 > 0 ) && (i2 > 0 )) {

if ( x1[i1] == x2[i2] ) { ++count; --i1; --i2; } else if (x1[i1] < x2[i2]) --i2; else --i1;

}

ie. 3

Multiple Choice Question 2, of 12 (a “fixed code” question)

Correct, 65%

Wrong, 23%

After the while loop finishes, “count”contains what value? a) 3b) 2c) 1d) 0

(Formatting compressed to fitquestion into this space.)

int[] x1 = {1, 2, 4, 7};int[] x2 = {1, 2, 5, 7};int i1 = x1.length-1;int i2 = x2.length-1;int count = 0;while ((i1 > 0 ) && (i2 > 0 )) {

if ( x1[i1] == x2[i2] ) { ++count; --i1; --i2; } else if (x1[i1] < x2[i2]) --i2; else --i1;

}

ie. 3

Multiple Choice Question 2, of 12 (a “fixed code” question)

But first, the preamble to the question …If any two numbers in an array of integers, not necessarily consecutive numbers in the array, are out of order (i.e. the number that occurs first in the array is larger than the number that occurs second), then that is called an inversion. For example, consider an array “x” that contains the following six numbers:

4 5 6 2 1 3 There are 10 inversions in that array, as:

Multiple Choice Question 8, of 12 (a “skeleton code” question)

x[0]=4 > x[3]=2x[0]=4 > x[4]=1x[0]=4 > x[5]=3x[1]=5 > x[3]=2 x[1]=5 > x[4]=1

x[1]=5 > x[5]=3x[2]=6 > x[3]=2x[2]=6 > x[4]=1x[2]=6 > x[5]=3x[3]=2 > x[4]=1

int inversionCount = 0;

for ( int i=0 ; i<x.length-1 ; i++ ) {

for ( xxxxxx ) {if ( x[i] > x[j] ) ++inversionCount;}

}… the “xxxxxx” should be replaced by:a) ( int j=0 ; j<x.length ; j++ )b) ( int j=0 ; j<x.length-1; j++ )c) ( int j=i+1; j<x.length ; j++ )d) ( int j=i+1; j<x.length-1; j++ )







Correct, ~50%






Correct, ~50%

The idiomatic loop






Correct, ~50%

Wrong, ~30%

The idiomatic loop






Correct, ~50%

Wrong, ~30%

The idiomatic loop


Students across all institutions (N=556), percentage correct for each of the 12 MCQs.

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12

Question number

per

cen

tag

e co

rrec

t

Q8, 3rd hardest

Q2, 6th hardest


Moral: a difficult question at one place is difficult elsewhere

Percentage Correct per Question by Institution with N>20

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12

Question No.

Pe

rce

nta

ge

Co

rre

ct


Quartile Score

Top 10-12

Second 8-9

Third 5-7

Bottom 0-4

12 universities!!! 7 countries!!!>500 students

(If you failed more than 25% of class)

Quartiles on the 12 Multiple Choice Questions

The third quartile determines the “bottom passing” student.


0

5

10

15

20

25

30

35

40

1 8 16 24 32 40 48 56 64 72 80 88 96

Scores

Nu

mb

er o

f S

tud

ents

Quartile Scores

Top 10-12

Second 8-9

Third 5-7

Bottom 0-4

McCracken et al., 2001

Lister et al., 2004


The Story So Far

• Performance Data

… and now …


The Story So Far


… and now …

• Doodles– Did students draw pictures and what

sort

“Doodles”


“Doodles”


Doodle Performance

Percentage of correct answers when students (N=56) use a particular doodle type on any question (12).

Doodle Category %Correct Data Points

… … …

Synchronized Trace (S) 77% 73

Trace (T) 75% 215

… … …

Blank Page (B) 50% 256


Blank Doodles

6th hardest, 65% correct 3rd hardest, ~50% correct


Doodle Discussion

• Students appear to lack good doodling strategies

• Should we teach doodling (see later)– Assess doodling?


The Story So Far


• Doodles

… and now …


The Story So Far


• Doodles

… and now …

• Interviews, “Think Out Loud”– Transcripts from 38 students


A transcript from Question 2This time I'm going to look to see what the question wants first. So trying to find the value of count. So now I'm looking to see what each variable is set to and what's happening inside the program. I'm going to go ahead and write down that x1's length is 4 and x2's length is 4. And then e loop, 4 is greater than 0 and 4 is greater than 0. So inside the loop, the, checking to see if the 5th, checking, noticing that I didn't subtract 1 from each of i1 and i2. So changing i1 to 3 and i2 to 3. So when the 3rd spot, last spot of each array, they're equal, so it enters the first if loop so count is now equal to 1; i1 equals 2 and i2 equals 2. Now we're going through the loop again. 2 is greater than 0 and 2 is greater than 0 so we're inside the loop. Checking the first "if" are the second indexes equal to each other? no. So going to the else if and ‘is 4 less than 5?’, which is true so i2 is subtracted 1. I2 now equals 1 and we're going back to the while loop again. 2 is greater than 0 and 1 is greater than 0. So checking the first if ‘is 4 equal to 2?’, which is false so checking the second if. “Is 4 less than 2?’, which is false so doing the else statement which is subtract 1 from i1. so i1 now equals 1. So going through the while loop again. 1 is greater than 0 and 1 is greater than 0. So checking the first if "is 2 equal to 2?" which is true so increment count by 1. Count is now 2. Subtract 1 from i1 which becomes 0 and i2 which becomes 0. So now the while loop fails because 0 is not greater than 0. So it asks for the value of count which is 2.


Possible Remedies?

• Other languages?

• Force students to produce doodles?

• Support for collaborative work?

• Technology?– Integrated Development Environments (IDEs)

For example, BlueJ (Kolling etc.), TogetherJ (Borland), …..


In particular we wanted:• support for scaffolding of key concepts • support for collaborative learning, and

• a way of collecting information on how our students actually approach the design and coding process

So, we decided to build our own


Four experiments at Aberystwyth

and how they have affected the design of our IDE – Vortex

1. Learning Styles

2. Scaffolding with Visual Materials

3. Pair Programming

4. Collaborative Design



1. Learning Styles

• Various approaches: left/right brain, Meyers Briggs, Kolb Learning styles, …

• Felder-Silverman model of preferred learning style.

• This identifies what is easy for the student. Felder believes that students need to improve other styles of learning too.


Active: try things out, work with others

Reflective: think things through, work alone

Sensing: concrete, practical, facts, procedures

Intuitive: conceptual, innovative, theories and meanings

Visual: pictures, diagrams, flow-charts

Verbal: written or spoken explanations

Sequential: incremental, orderly steps

Global: holistic, learn in large leaps

Felder believes that all undergrad

education should be inductive

(specific to general) not deductive


Active: try things out, work with others

Reflective: think things through, work alone

Sensing: concrete, practical, facts, procedures

Intuitive: conceptual, innovative, theories and meanings

Visual: pictures, diagrams, flow-charts

Verbal: written or spoken explanations

Sequential: incremental, orderly steps

Global: holistic, learn in large leaps


Experimented and concluded…Some (small) evidence that Active and Visual

(biggest groups) learners needed more support

Courseworkmark

Exam mark

n

Overall 63% 56 % 107

Active 61 % 52 % 58

Visual 61 % 54 % 82


So, what did that mean for Vortex?

• First, all first year students are encouraged to reflect on their learning style and pointed to types of materials

• In the design of Vortex, we wanted to:– Add visual material, and – Encourage active learning


2. Scaffolding with Visual Materials

person1

Fred

Aberystwyth

person2

Bill

Borth

Person person1 = new Person(“Fred”,

“Aberystwyth”);Person person2 = new Person(“Bill”, “Borth”);person2 = person1;


person1

Fred

Aberystwyth

person2

Bill

Borth

Person person1 = new Person(“Fred”,

“Aberystwyth”);Person person2 = new Person(“Bill”, “Borth”);person2 = person1;

person1.setAddress(“Llan”);------------------What is person2’s address?


What we wanted

• To encourage our students to draw Object Diagrams

• How? By providing them with partially completed diagrams as scaffolding.

• Ultimately add this facility to Vortex?


Research Questions

• Is drawing some kind of Object diagram correlated with success in solving multiple-choice tracing questions?

• Does providing students with scaffolding in the form of partially completed Object diagrams help them correctly answer multiple-choice tracing questions?

• Do students who have been provided with this scaffolding continue to use it in such multiple-choice questions?


Is drawing some kind of Object diagram correlated with success in solving multiple-

choice tracing questions?

Group

n Follow-up Test Avg - Just on tracing

questions

Beginners who do not use diagrams

23 47%

Beginners who do use diagrams

45 68%

The answer is a (very) qualified yes (compare Lister)


Does providing students with scaffolding in the form of partially completed Object

diagrams help them correctly answer multiple-choice tracing questions?

We thought that the answer to this would be an obvious ‘yes’ but it appears to be ‘not really’

36% for people given diagrams 28% for those not given diagrams


Do students who have been provided with this kind of scaffolding continue to use it?

Not very surprisingly in light of last question, ‘no’


Why?

In Hegarty and Narayanan model, the viewer:• decomposes the system into simpler components, • constructs a static model by making representational

connections to prior knowledge and other components,

• integrates information between different representations (e.g. text and diagrams),

• hypothesizes lines of action, and finally • constructs a dynamic mental model by mental

animation.


Why?

In Hegarty and Narayanan model, the viewer:• decomposes the system into simpler components, • constructs a static model by making representational

connections to prior knowledge and other components,

• integrates information between different representations (e.g. text and diagrams),

• hypothesizes lines of action, and finally • constructs a dynamic mental model by mental

animation.

We did

They did?


• We short circuited the process? Or,

• When you are really lost the diagram looks like just one more thing to understand!



• We are trying harder to give students more practice in drawing object diagrams themselves.

• But in the design of Vortex, we are holding off on adding this facility.


3. Pair Programming• “a programming technique where two people

program with one keyboard, one mouse and one monitor” [Beck]

• Part of the XP set of techniques• Good results! (Williams and Kessler)

– Programs passed 15% more of test cases

– 95% of students agree “I was more confident in our assignments because we pair programmed.”

– 84% of the students report enjoying the experience of programming more when working in a pair


Would it work with our students?

• 18 year old males straight out of school.• We do a lot of team-building exercises in

the first year and feel that many of our students need them!

• It often seems to be the students who have better social skills who are the ones who lack confidence in their programming ability.


The Two Extremes

Code Warriors I have had no trouble at all completing programming tasks to

date, in fact they weren't challenging enough. I love to program and anticipate no difficulty with this course.

Code-a-phobes I don't like programming and I don't think I am any good at it.

I can write simple programs but have trouble writing new programs for solving new problems.


Fig 1: Student Distribution

warrior - 17

phobe - 13

middle - 34


Results

• Ability vs. Confidence• Performance in the course was not that different

Average for whole class: 59.7%

Average for Warriors: 62.8%

Average for Phobes 55.7%

None statistically significant


Strategy for Experiment

• Idea introduced in lecture

• Students then given brief paper to read

• Then experiment in regular 2 hour closed lab situation

• Done twice – With ‘opposite’ partner (at extremes)– Then again with ‘similar’ partner


Results of ‘opposite’ experience

• Overall, students enjoyed the experience (66%) and thought that it helped them produce a better solution (66%).

• But only 53% of the warriors reported enjoyment of the experience and only 47% of them thought that pair programming led to a better solution (statistically significant).


Achievement

• totally opposite pairs – 11 2.23

• somewhat opposite pairs – 8 3.19• similar pairs – 13 2.69


Results of ‘similar’ experience• Overall: 64% enjoyed experience and 65%

thought it led to a better solution• Warriors: 58% of them enjoyed the experience

and 67% thought it led to a better product • Overall: 44% of the students reported liking the

second experience more than the first (and 38% just as much).

• Phobes: 60% enjoyed this experience more, and none less, than last time.


Relevant Conclusions• Overall, first-year students like pair programming

and believe that it helps them achieve good solutions.

• The students whom we have identified as ‘warriors’ like pair programming the least.

• Students with less self-confidence seem to enjoy pair-programming the most.

• Students produce their best work when paired with students of somewhat, but not very, different levels of confidence. (some evidence)



• We confirmed that many of our students liked and seemed to learn more when they could work in groups.

• The theory that collaborative work is best done in mixed ability groups seems to have some validity for our students.

• Design goal of basing VorteX around support for collaborative working was valid.

• Vortex should encourage the pair programming idea that pairs ‘talk’ about what they are doing and why. (This was the most frequently mentioned advantage.)


VorteX tool has indeed been constructed

To provide a fully interactive, collaborative development environment that enables the capture of novice design knowledge and gives structured feedback to educators.

Student explanation for decisions give three benefits: • they generate in-line documentation; • they force the users to think about each action they

are performing (see PP experiment); and • they provide the instructor with valuable data

about student misconceptions.


Collaborative Design

Considerable evidence that collaborative work is helpful, but not without its difficulties.

Slavin suggested three qualities for successful collaborative working projects:

1. team reward,

2. individual accountability, and

3. equal opportunities for success



4. Experimenting with Vortex

• This experiment is different because we now have the tool.

• Groups in lab used Vortex for a design session

• Randomly assigned groups


Feedback from using Vortex

• “I found the experience both enjoyable and insightful.”

• “It made it easy, I could make stupid mistakes anonymously.”

• “If a decision is reached about two different ways of doing something and the group is split on the matter, it is more difficult to resolve, but being anonymous makes it easier to say what you need to.”


Observations

• Every group (except the one that met face to face!) rated the exercise as 'very positive'.

• Many groups never met face to face even at the end of the lab.

• Students with disabilities were not disadvantaged (as often happens).

• Much more peer support and less need for tutors.



• We have confirmed that working collaboratively through Vortex is useful.

• We have addressed Slavin’s points.

• We have collected a huge amount of data to analyse.

• The enthusiasm for anonymity was unexpected and interesting and needs to be investigated further.


Three original design goals

• support for scaffolding of key concepts • support for collaborative learning, and • a way of collecting information on how our

students actually approach the design and coding process

We have accomplished last two and provided an active learning environment with some visual support


Reflection and Conclusions

• Process has perhaps been more important than what we have done.

• It is easy to decide what is right for students based on what is right for you.

• A different mindset has been established in staff:

Understanding students is the key to success!

Documents

Johnny can’t Program (and neither can Johann) Lynda A. Thomas Department of Computer Science, University of Wales, Aberystwyth, Wales, UK