Student EvaluationsDo lecturers think they are worthwhile and use them to enhance student learning?

2011

Dorothy SpillerSenior Lecturer, Teaching Development Unit, University of Waikato

Lecturers’ Perceptions?• Are students able to judge the

quality of the teaching and learning?

• Do lecturers engage with and act on student evaluation feedback?

• What other factors influence lecturers’ willingness to engage with student feedback?

2

Context: The Research Project

• Two year research project funded by Ako Aotearoa (New Zealand Centre for Tertiary Teaching Excellence)

• Three collaborating institutions

3

Goals of the research project

• Aim to provide evidenced-based information about the impact of teaching evaluations on lecturers’ thinking and behaviour at all stages of the teaching and learning process

• Aim to make recommendations for institutional policies and practices around student evaluations that can enhance the developmental benefits of evaluations for lecturers and improve students’ learning experiences

4

Research Methods

• On-line questionnaire conducted at all three institutions that provided quantitative and qualitative data

• Response rates: 47.19% (Otago University); 37% (The University of Waikato), 45.42% (Otago Polytechnic)

• Interview questions developed from themes that emerged in the questionnaire

• Twenty interviews conducted at each institution

5

The literature

• Reliability of student comments questioned-many frequently reported views, but not much extensive empirical evidence. Common views include that students are too immature, that a lapse of time is needed before judgement is possible, and that there are too many irrelevant variables such as lecturer popularity

• General teacher attitudes. Popular view is that lecturers hate student evaluations, but considerable evidence to challenge this (Braskamp & Ory, 1994; Nasser & Fresko, 2002; Penny &Coe, 2004; Beran & Rokosh, 2009)

• Beran and Rokosh (2009) caution that the mildly positive views do not correlate with actual usage of instrument for teaching improvements

6

Literature

• Possible link between teaching and learning beliefs and response to evaluations (Hendry, Lyon and Smart, 2007)

• Teacher emotions and response to student evaluations (Moore & Kuol, 2005; Arthur, 2009)

7

Waikato Interviews: Findings

Group 1: Emphatically agreed that students can make judgements

7 (out of 20)

eg. “Too damn right, they are eminently well placed to judge” (J)

Teaching Beliefs: 6 out of 7 held student-focussed teaching conceptions

eg . “to facilitate the most profound learning that I can” (J)

Engaging and responding: 6 out of 7 engaged and used feedback to modify teaching

eg. “They are really important to me…I take these comments and use them as objectives for myself of things that I need to change or adjust”

8

Waikato Findings

Group 1 : Emphatically agreed students can make judgements

Emotional factors

• Generally acknowledge emotional dimension but try to manage it

Other factors

• Timing of the evaluations

• Concerns about statistical reliability of results

• Perception that students complete forms in a rush

9

Waikato Findings

Group 2: Agree with reservations that students can make judgements

(10 out of 20)

eg. “Yes [but] You can be a good teacher, but if they haven’t understood it-that can be frustrating” (D)

“ Yes [but] most students are not interested in doing them, it’s just another tick thing” (F)

Teaching beliefs

Not a consistent pattern here but overall less evidence of depth of reflection about teaching and learning than the first group. 5 out of 10 express beliefs from standpoint of the teacher

10

Waikato findings: Group 2

Engaging and responding

• Most of this group engage to some degree with student feedback

• Only I academic in this group ignores student feedback

Emotional factors

Some evidence of emotions in language of some respondents, although emotion not specifically addressed by these respondents.

eg. of emotive language- “rip open the packet” “frustrating” “pretty disappointed” “I can’t trust them”

Other factors

Problems with the questions (3) timing (4) use for promotions (3)11

Waikato Findings

Group 3: Resoundingly negative about students’ ability to make judgements

(3 out of 20)

eg. “No, I don’t think the mark connects with reality”

Teaching beliefs

•Very limited responses, all teacher-centric

Engaging and responding

•One considers quickly, one “a quick glance” and one completely ignores

12

Waikato FindingsGroup 3: Resoundingly negative about students’ ability to make judgements

Emotional factors

Emotional language most evident in this group

“eg. I read them, then I think now what did I do to you again?”

“terrifying” “some responses are personally cruel”

Other factors

•Students have consumer mentality

•Use for promotions-wrong ethos

•Timing

13

Discussion

Generally positive disposition towards appraisals supported by literature (eg Penny & Coe, 2004; Beran & Rokosh, 2009). Largest group is those who agree but with reservations, supports the finding of Nasser & Fresko (2002) who found teachers in their study to be “mildly positive”.

In relation to engagement and use of evaluation feedback, this sample of teachers were more positive than those in the studies of Beran & Rokosh (2009) and Smith (2008) who both query whether fairly positive views translate into actual use of evaluation responses to modify practice. In our sample, 15 out of 20 teachers engage with student feedback to varying degrees.

Cautions; one institution; potentially biased sample.

14

Discussion of Waikato findings

There was not a consistent alignment between particular teaching beliefs and views of students’ evaluation feedback as found in the study of Hendry et al (2007). However there were some noticeable patterns. Those most emphatically affirming student feedback were very reflective about teaching and learning and student-focussed in their observations (6 out of 7). In the group who agreed with reservations, there was not the same degree of reflection as in the first group, and they were evenly divided between student-focussed thinking and teacher-centred perspectives. With the resoundingly negative group, there was almost no sign of reflection and hey all discussed their teaching from a teacher’s point of view.

15

Discussion of Waikato findingsOther factors influencing engagement

Recurrent themes in all groups- timing, use of instrument for promotion and the more general observation that the university does not value teaching. These findings concur with concerns expressed in a number of studies about the ways in which student evaluations are used by management (McKeachie, 1997; Penny & Coe, 2004; Beran & Rokosh, 2009).

Emotion

This did not feature nearly as powerfully as it did in the questionnaire responses or as suggested by Moore and Kuol (2005) and Arthur (2009). Only one institution’s results. Possibly the question was not well-designed.

16