Module Evaluation Toolkit June 2012

Student Evaluation – Toolkit for Module Teams

Introduction

This toolkit aims to support enhancement of student learning and the development of a partnership between students and staff, with each having a role in further developing learning and teaching. It provides ideas for how to approach a shared evaluation of student learning experiences.

Teesside University has a Threshold Quality Standards for Student Evaluation. The TQS underpins the evaluation approach at Teesside. The TQS outlines the requirements for evaluating the student learning experience at each level and across all modes of delivery. This toolkit provides support for one aspect of this evaluation process.

This evaluation toolkit is provided as guidance, and its main focus is to provide a set of tools you can use or adapt for evaluating teaching practice, the student learning experience, and student learning in a variety of settings and contexts. It also briefly explores the why, what, when and how of evaluation: why evaluate; what to evaluate; when you should evaluate; and how you might do the evaluation. The section on how you evaluate includes suggestions on some of the other material you might want to draw on when you are evaluating a particular learning context. There is also an important closing section to this toolkit, closing the loop, which provides guidance on how to use what you learn from any evaluation in future development of practice.

WHY evaluate?

To:

o Provide an opportunity for students to give feedback to the tutor that teaches them

o Find out what didn’t work

o Identify what worked well

o Highlight innovative/good practice

o Decide what to keep or to change to impact on student success

o Provide material for the tutor in support of reflective practice

o Provide professional satisfaction for staff

o Provide evidence to argue for changes e.g. learning accommodation or equipment

o Because it is a requirement as part of Annual Review and Monitoring (ARM) to evidence standards

1

Module Evaluation Toolkit June 2012

WHAT to evaluate?

The focus should be on the evaluation of the student learning experience, which may encompass one or all of the following:

The learning process

The outcome of the learning

The relevance of the learning - to the discipline, student employability etc

Be clear about what do you want to know and what you will do with the information obtained.

WHEN to evaluate?

Timing of evaluation is also important. This may be determined in some cases by institutional requirements e.g. end of stage/year or new module evaluation e.g. Teesside TQS – Student Evaluation.

However, these are minimum requirements and may not represent the most appropriate strategy for your context. Evaluation should be timely, not over-cumbersome, relevant, and create opportunity for identifying both what works and what needs to change.

Other additional or alternative options include:

Evaluating during, as well as, or instead of at the end of a module

Evaluating at a key point in the module

Evaluating in response to a particular area/issue, so for example if attendance is dropping off you might want to look at some sort of in-module evaluation and then adapt the L&T strategy/individual sessions accordingly

Providing an ongoing open invitation for feedback - e.g. through a feedback section on the VLE (e.g.Blackboard) site – you would need to monitor this regularly AND respond so that students can see that their feedback has been taken account of and what impact it has had

Providing regular evaluation opportunities e.g. post-it note/flip chart ‘question of the week/fortnight’ (see ‘evaluation ideas’ below)

HOW to evaluate?

What is a useful way to access information to evaluate the module, or the learning context you are exploring? The method used should be: relevant to students; interesting enough for them to participate; and produce outcomes they will recognize if possible, so, for example you might use a ‘you said...we did’ approach.

2

Module Evaluation Toolkit June 2012

While you will want to ask your students about their experience in some way, and this tool-kit provides a range of examples of how you might do this, you might also want to draw on information from other sources, so for example,

Other Information:

A range of useful module evaluation information may already be available for use, for example:

Individual staff/module team reflections i.e. what worked, what didn’t work, ideas for enhancement etc. Some of this information may be, for example, in programme team or section meeting notes.

Statistics on student performance, progression and completion

Attendance data

o You may need to drill down to determine causes of poor attendance e.g. assessment dates, correlation with other modules

o Research on attendance suggests that students will generally attend sessions that they perceive as relevant to their programme, beneficial in some ‘added value‘ way, and interesting.

External Examiner reports

Student Representative feedback

Peer review of teaching/informal professional conversations between staff

You will however also wish to generate specific information to help you evaluate the module or other learning context, and this is where the following ideas for evaluation methods should be of use. A variety of ideas are presented for you to choose from/adapt or develop into your own method. The aim is to help you identify the best approach to suit your module and context. A crucial first step is to decide if you are evaluating the learning process (how students are learning), the outcomes (what have they learned), or both. You might use a different method depending on this decision.

Evaluation ideas:

When including the suggestions identified below in this toolkit, the aims were to:

Promote positive change by including methods that encourage positive rather than negative feedback and actively promote 'suggestions for change' rather than purely critical comment.

Encourage staff/student dialogue by providing staff and students with the means to open or engage in a productive and informative dialogue as a result of and in reply to student feedback.

Address the issue of trust by highlighting clarity of purpose and use of student feedback and evaluation.

3

Module Evaluation Toolkit June 2012

Focus on easily adaptable ideas by focusing on format and methodology rather than any one particular subject or discipline area.

Focus on easily implemented ideas with a minimum of administration and evaluation.

Offer a range of options that would be applicable to a range of different needs and approaches.

A number of the ideas outlined below can be adapted or used in various ways. So for example they can be used: on an individual, small, or large group basis; they can be used in conjunction with ‘snowballing’ and teacher/student clarification and discussion; they may be used to evaluate student learning outcomes, or the learning process. Equally, many of the methods can work well during the module and/or at the end of a module, or they might be used in an ongoing way.

Text based

1. Collect one word/one sentence responses to module using learning related triggers e.g. ‘my most important learning from this module.....; the least useful.....; one thing I will do as a result of this module.....etc. These can be easily collected using:

Feedback section on module or programme Bb site Interactive whiteboard during session Post it notes during a session

2. Reflection form submitted with the end of year assessment (possibly as part of specific assessment criteria) which includes a small number of questions on their performance in assessment?

3. ‘Start, stop, continue’ – Students are asked to consider what they will start doing, stop doing, or continue doing in the light of what they have learned on the module. Students write their responses on post-its or specific handout (see Appendix for templates).

4. Engage current students in producing guidance or resources for subsequent cohorts e.g. create the whole, or elements, of the module or programme handbook; develop a ‘top tips’ for learning a particular topic or succeeding in the assessment etc. This engages students as co-producers and it may be possible to incorporate this type of activity into the module assessment strategy.

5. Students complete a one-sentence summary in response to a module evaluation question. For example: their experience of the module; the effectiveness of the module in supporting their learning etc (One Sentence Summaries p183 *).

6. Students identify a single word to represent their evaluation of the module/session/topic and then a few sentences or bullet points to explain why he/she chose that word. (Word journal p188*). If used as a regular activity this could support personal development planning (PDP)..

7. Provide students with an outline of key elements of the module content/experience for which you want feedback. Students fill in the outline to provide the detail/key learning points/highs and lows of their learning/experience concerning each element (Empty Outlines p138*).

4

Module Evaluation Toolkit June 2012

8. Students identify their learning goals at the start of the module, rank these in terms of importance to them and also potentially their estimated relative difficulty in achieving these. These can then be matched against the tutor’s learning goals for the class followed potentially by discussion and clarification regarding how and why there may be mismatches. The results of this activity can be revisited and revised midway, or at the end of the module as a means of student and staff evaluation of the module process and outcomes (Goal Matching and Ranking p290*).

9. The teacher quickly polls students on 3 questions related to their learning . These may be worded differently but basically include: ‘What works?’, ‘What doesn’t?’ and ‘What can be done differently to improve it?’ The most frequent responses are identified, summarized and then used by the tutor to enhance the learning experience. This technique allows the tutor to see the session or module through the students’ eyes. (Group Instructional Feedback Technique [GIFT] p334*)

Captured audio/video ideas

10. Audio feedback – set up recording equipment and invite students to do verbal or video feedback – this could be constructed as a ‘think aloud’ exercise – what I’ve learned, what I would have liked, what I achieved, what worked and what didn’t.

11. Student diaries – written/audio/video collected at various points throughout the module.

Verbal/discussion

12. Ask a question of the group every couple of weeks to map module progress and success of learning and communication. This is useful for staff but also helps students see that they are learning and changing. In particular it helps students to appreciate the value of the ongoing learning experience and shows that you are responding to at least some of their concerns if you use a ‘You said – we did’ format.

13. Peer feedback – students interview each other about what they got out of the module.

14. Student Union, Retention Support Officers or course reps come in and do a focus group then provide you with feedback.

15. Use the seminar/workshop exercise (available on the PDP support site in Blackboard– in the section on reflection) – staff get feedback and students gets material for their personal development planning. Some other reflection based exercise can be used in this way.

16. Cumulative round table feedback activity– students work in small groups to very quickly brainstorm all the strengths of the course module and then all the things about the course that could be improved using the proforma in Appendix. The paper circulates from one student to another quickly as each writes down their idea and says it aloud. These brainstorms are done very quickly to prevent

5

Module Evaluation Toolkit June 2012

students falling into analytical mode. Then each group rank orders the strengths and areas for development they have identified using a process of consensus. If time allows, each group can then share their top three strengths and areas for development (see Millis, 2004 p58).

17. Nominal group technique – students work initially individually to complete the proforma in Appendix. They then pair up with a peer to develop a joint response to the proforma adding points to each section that they both agree on. Each pair then joins with another pair and the process is repeated. This cycle can be repeated a number of times to be determined by the size of the group before responses from each group are shared and added to a whole group proforma. The benefit of this specific proforma is that it encourages joint responsibility for the learning experience (Adapted from Eastcott 1992).

18. Students raise their hands to indicate agreement or disagreement with specific module evaluation statements and issues. This encourages students to share their own opinions but also to compare these with those of their peers. (Opinion Polls p258*). An opinion poll can also be done using mobile technology, e.g. using Poll Daddy or other online tool (for help with using mobile technology to support teaching and evaluation talk to a member of the e-learning team in DfLD)

19. Students identify the most meaningful or most difficult aspects of the module. They then explain the personal significance of the module for them and their response to the aspects previously identified. This engages the students in a dialogue with their evaluation of the module and their reactions to this learning experience. (Double Entry Journals p263*)

20. In small groups, students develop a ‘news flash’ or a ’sound bite’ for other students who might study this module highlighting why it’s useful, or why it isn’t, and who it is useful for, or who it wouldn’t suit.

Visual/creative

21. Students ‘draw’ their experience e.g. as a journey, choose a metaphor etc then explain/discuss.

22. Students could use words from one of the text options above to develop a narrative or short poem about the module.

23. Students use ‘green’, ‘amber’ or ‘red’ cards/paddles to provide a visual or virtual evaluation of responses to evaluation questions.

24. Students/small groups choose a picture postcard or artifact e.g. postcard from a selection provided or of their own choosing to represent their experience of the module then explain/discuss.

25. Small groups ‘act out’ their module experience as a story/live performance.

26. Students complete the second half of the analogy e.g. ‘This module is to my learning what......is to......’. For example: this module is to my learning what:

...salt and vinegar are to chips

....food is to a starving man

6

Module Evaluation Toolkit June 2012

.....a lifebelt is to a drowning man

....Chinese is to an English speaker

(Approximate Analogies, p193 *).

27. Students make a drawing or diagram of their learning from /experience of the module in the form of a concept map. The map should include important features of the module for them e.g. a mountain range may indicate the highs and low points which they should be encouraged to identify, a communications centre depiction may be used to depict success or otherwise of communication methods used during the module e.g. with module team, peers, a group task etc) (Concept Maps p197*).

28. Give students a list of topics covered/aspects of the module and a very limited time to rank these against a given statement e.g. easiest to grasp to hardest to grasp. Individual rankings can be added to a large scale display on the wall with students using different coloured stickers to identify their rankings as part of a collective overview. This could be adapted further using a screen version and voting handsets, or a mobile voting system (like Poll Daddy or Poll Anywhere) for students/small groups to indicate their ratings. (Categorising Grid p160*)

Online

29. Use a student blog tool in the module – This can support students PDP development AND staff get valuable feedback – you could consider incorporating this as part of the module assessment where appropriate.

30. Text or tweet responses to question during a session? – could potentially do this as a visible conversation.

31. Use a mobile polling tool (e.g. Poll Daddy; Poll Anywhere). You identify questions in advance, put them into the online poll and during the class students text (or tweet) a code response to a number to register their vote. This can be used to gauge students understanding or to ask for views on aspects of the module. Where not all students have mobile phones, students can be grouped to agree a response and one of them, with a phone, texts the answer on behalf of the small group. For help using online tools or mobile technology talk to a member of the elearning team in DfLD

Quantitative

32. Students rate their self- confidence regarding specific elements of the module e.g. relevant knowledge, skills etc. They can then be asked to identify what helped or hindered the development of their confidence during the module and why. (Course Related Confidence Activity p278*).

33. Online feedback options – e.g. survey monkey 3 questions for students to explore outcomes (not same as normal survey method reserved for end of year/stage survey).

7

Module Evaluation Toolkit June 2012

*adapted from Angelo and Cross (1993)

CLOSING THE LOOP

Don’t forget that collecting evaluation data is only part of the story, evaluation is a process,

The process is:

The module evaluation process is a potentially important part of supporting programme/course teams and subject groups in enhancing what they do. Here are some ideas on how you might use it with colleagues.

8

PLANdecide on an evaluation

method/s which suits your needs and context. Set

objectives/ask questions/make predictions.

DOcollect evaluation data (don't forget other sources of data,

e.g. pattern of assessment marks, retention statistics).

STUDYsort and reflect on what comes out of the evaluation - this can be done during the module or

at the end. It can be continuous or at specific

points.

ACTdo something with what you

found. Use it in annual review and monitoring, but also use it

to make change/share good practice and show how

students respond to their feedback.

This stage is particularly important. It can:

provide an opportunity to encourage feedback from the current student group

illustrate how student feedback has been used

provide students with a sense of true partnership in the evaluation process

Module Evaluation Toolkit June 2012

Module evaluation activities and review can be used in conjunction with peer review of teaching to model and share good practice and/or to help each other resolve challenges you might face (for example in communicating difficult concepts, or dealing with a disruptive or passive group of students)

Programme teams can have evaluation discussion sessions annually (or termly?) where each member of the team shares their evaluations. This can provide a better understanding of the students on the programme, or of a particular cohort, and of their overall learning.

Sharing evaluations with colleagues can provide support. Where a particular issue has arisen, discussing this with colleagues can help you find a solution.

Sharing evaluations with the programme team or subject group can enhance everyone’s practice by identifying what works, and what doesn’t.

Sharing evaluations can provide guidance and support for less experienced colleagues. Hearing about other people’s teaching experience and what evaluations said can

reinvigorate our own ideas about teaching. Collating material from evaluations of modules within a programme can provide material for

revalidations and ideas for new programme developments. Discussing evaluations from modules which are shared across multiple programmes can

help with understanding different discipline approaches to learning, and can enhance cross programme practice

It is worth exploring within the programme team how you might use your module evaluations and the evaluation processes to more effectively share good practice and improve the student learning experience on the programme.

Final Word

Thanks to all those staff at Teesside and beyond who provided examples of tools for evaluation, or who read and commented on this toolkit. This is an ongoing project, so if you have any other suggestions for tools or additional resources, please contact diane.nutt@tees.ac.uk or g.janes@tees.ac.uk

ADDITIONAL RESOURCES

From QAA Scotland:

http://www.enhancementthemes.ac.uk/student_evaluation/index.html - this is a rich source of further ideas as well as more detailed information on the Why, What, When and How of student evaluation and feedback. It includes an extensive bibliography and a number of links to other useful resources

From the Association of Learning Technologists (ALT):

https://sites.google.com/site/evaluationtoolkit/ - this resource is based on the six steps to effective evaluation available at: http://www.jisc.ac.uk/media/documents/programmes/digitisation/SixStepsHandbook.pdf

9

Module Evaluation Toolkit June 2012

10

Module Evaluation Toolkit June 2012

REFERENCES

Angelo TA and Cross KP (1993) Classroom Assessment Techniques: A Handbook for College Teachers(2nd Ed) San Fransisco Jossey Bass

Eastcott D (1992) Evaluating Your Teaching: Further Induction Pack IV SCED Paper 68 Birmingham, Standing Conference on Educational Development

Millis, 2004 Using Interactive Focus Groups for Course and Programme Assessments, in Swing, R.L. (ed) Proving and Improving Volume II: tools and techniques for assessing the first college year, (Monograph, no 37) Columbia, S.C. USA: University of South Carolina, National Resource Center for the First Year Experience and Students in Transition

This Toolkit was written by

Gill Janes and Diane Nutt (Department for Learning Development)

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License: http://creativecommons.org/licenses/by-nc-sa/3.0/

11

Module Evaluation Toolkit June 2012

Appendices:

Stop-Start-Continue templates

12

Module Evaluation Toolkit June 2012

13

Module Evaluation Toolkit June 2012

Nominal group technique proforma (Eastcott, 1992)

Good Points Areas for Enhancement

About the COURSE/MODULE

About the TEACHING

About us, the STUDENTS

14

Module Evaluation Toolkit June 2012

Round table activity template(Mills 2004p58)

Activity 1

Passing this sheet of paper rapidly from one person to another, please jot down all of the relevant strengths of the course/module, saying them aloud as you write.

Working as a team, rank order the strengths you identified, with the most important ones at the top of your list. Rank at least three by writing numbers ‘1’, ‘2’ and ‘3’ next to the strengths you identified.

Activity 2

Passing this sheet of paper rapidly from one person to another, please jot down all of the ‘negatives’ of the course/module – the things you would change, saying them aloud as you write.

Working as a team, rank order the weaknesses you identified, with the most significant ones at the top of your list. Rank at least three by writing the numbers ‘1’, ‘2’ and ‘3’ next to the weaknesses you identified.

15