What makes a good evaluation? - University of Sheffield/file/MasterclassEvaluation.p… · Types of...

What makes a good evaluation?

Write your answer on a post-it

Public Engagement Masterclass 30: Some

ideas for evaluating your public engagement activity

Fran Marshall, Research & Evaluation Manager Dr Helen Featherstone, University of Bath

27/01/2016 © The University of Sheffield

Outline of session

• Evaluation aims, planning and methods – Fran

• Some examples of evaluation - Helen

Introduction to the project and Public Engagement and Impact Team

• Introduction to the Public Engagement and Impact Team

• The masterclass series

• What we can do to help?

27/01/2016 © The University of Sheffield

Masterclass series Masterclass title Date & time

20: Some ideas about how to evaluate your public engagement event Thursday 22 January 2015

21: What to think about when organising your public engagement event Tuesday 24 February 2015

22: Working in partnership with others

Wednesday 25 March 2015

23: Coordinating large scale public engagement, e.g. faculty based events, film or lecture series etc

Wednesday 29 April 2015

24: Public Engagement or Widening Participation/ Outreach? Things to think

about when organising events for children and young people

Wednesday 27 May 2015

25: Grant writing and impact Wednesday 24 June 2015

26: Writing a risk assessment for your event/ event planning Wednesday 29 July 2015

27: Engaging the public as researchers and different types of engagement Wednesday 30 September 2015

28: How to communicate your event

Wednesday 28 October 2015

29: Storytelling in public engagement Wednesday 25 November 2015

30: Some ideas about how to evaluate your public engagement event Wednesday 16 December 2015

27/01/2016 © The University of Sheffield

• The University’s Good Research and Innovation Practices policy

www.sheffield.ac.uk/ris/grip

• Minimum accepted practices in public engagement

• Higher practices to aspire to

27/01/2016 © The University of Sheffield

What makes a good evaluation?

Write your answer on a post-it

Who is here?

A. Academic staff

B. Non-academic staff

C. Post-graduate student

D. other

27/01/2016 © The University of Sheffield

Academ

ic st

aff

Non-aca

demic

staff

Post-g

raduate

student

other

14% 14%

38%34%

Which faculty are you from?

A. Arts & Humanities

B. Engineering

C. Medicine Dentistry and Health

D. Science

E. Social Sciences

F. Cross faculty

27/01/2016 © The University of Sheffield

Arts &

Hum

anities

Engineerin

g

Medici

ne Dentis

try a

nd H...

Science

Social S

cience

s

Cross

facu

lty

10% 10%13%

37%

3%

27%

How much do you know about planning evaluation?

A. A lot

B. Quite a lot

C. A little

D. Don’t know anything

27/01/2016 © The University of Sheffield

A lot

Quite a

lot

A little

Don’t kn

ow a

nythin

g

0%

32%

55%

13%

How much do you know about evaluation methods and when to use them?

A. A lot

B. Quite a lot

C. A little

D. Don’t know anything

27/01/2016 © The University of Sheffield

A lot

Quite a

lot

A little

Don’t kn

ow a

nythin

g

9%

34%

53%

3%

How much do you know about different kinds of evaluation?

A. A lot

B. Quite a lot

C. A little

D. Don’t know anything

27/01/2016 © The University of Sheffield

A lot

Quite a

lot

A little

Don’t kn

ow a

nythin

g

6%

32%

55%

6%

What is Public Engagement?

• “Public engagement describes the myriad of ways in which the activity and benefits of higher education and research can be shared with the public. Engagement is by definition a two-way process, involving interaction and listening, with the goal of generating mutual benefit”

National Coordinating Centre for Public Engagement

What is impact?

In relation to the Research Excellence Framework and Pathways to Impact:

• A change created by your research outside of academia

• From a small change in one person’s attitude to complete revolution in the way of life of millions worldwide

• Undertaking evaluation can help you demonstrate the impact of your project

27/01/2016 © The University of Sheffield

Where to start with evaluation…

27/01/2016 © The University of Sheffield

What is evaluation? • About the audience – show of hands

• Who has organised a public event before?

• Who has devised an evaluation?

27/01/2016 © The University of Sheffield

Monitoring or evaluation?

• Monitoring is counting

• Audience numbers, demographic information

• Evaluation is about critical reflection

• Can be: formative, about a process or summative

27/01/2016 © The University of Sheffield

When do we evaluate?

• Ideally before, during and after an activity

• In reality often done at the end - OK for single event but not for longer piece of research

• Includes looking at the quality of content, delivery and impact on participants

27/01/2016 © The University of Sheffield

Types of evaluation

• Formative: to support your activity, e.g. identifying gaps or barriers, gathering information to identify the need for your activity. Testing ideas and concepts – what is going to be most effective? What works best?

• Process evaluation: gathering information about how you can do it better next time

• Summative: i.e. a summary – final impact of your activity

• Is it a single evaluation or do you want base line and comparative data?

27/01/2016 © The University of Sheffield

Evaluation Tree

27/01/2016 © The University of Sheffield

What do you want to know?

27/01/2016 © The University of Sheffield

Start at the bottom

• Aims and objectives of event

• What kind of evaluation is it?

• What do you want to know?

27/01/2016 © The University of Sheffield

What do we need to evaluate? • Simple answer is the

public engagement activity you are doing

• More complex answer is what can we evaluate?

• What do you want to know v what can the audience tell you?

27/01/2016 © The University of Sheffield

Developing an evaluation plan

27/01/2016 © The University of Sheffield

Evaluation plan • Who do you want to evaluate?

• Who are your public?

• What resources are available for evaluation?

• What type of event is it?

• What questions should you ask?

27/01/2016 © The University of Sheffield

Who do you want to evaluate? • Audience

• Event leaders (University staff)

• Partner

• Volunteers

27/01/2016 © The University of Sheffield

Who are your ‘public’?

• Who are you expecting to attend your event?

• Design the evaluation activity around your audience

27/01/2016 © The University of Sheffield

Resources • Who will do your evaluation on the day?

• How much time do you have to analyse the data?

• What do you want to do with the data?

• How much money do you have for evaluation?

27/01/2016 © The University of Sheffield

Example costs

• Postcards - £35 for 1000

• A1 poster - £17

• Software for tablet survey - $49 per month

• Comment book - £10

• Google forms – free

• Surveymonkey - £25 per month

• Photographer - £75 per hour

• Plus pens, post-its etc

27/01/2016 © The University of Sheffield

What type of event is it?

27/01/2016 © The University of Sheffield

• Keep it simple and clear

• Should link to what you want to know

• Have a bottom line

• Ask the right questions

• Remember the unexpected

• Reflect on what you found out

27/01/2016 © The University of Sheffield

What questions should you ask?

27/01/2016 © The University of Sheffield

Asking questions

• Nothing is perfect – decide what fits best

• Keep it relevant

• Make it fun

• Don’t make it too onerous

• Put yourself in the participants' shoes

Evaluation questions exercise

1. How much do you earn?

2. Would you come to this venue again?

3. Do you have any problems with your boss?

4. To what extent do you agree - the Earth is like a spaceship with very limited room and resources?

5. Do you think URMS is a important method for calculating the fEC of the PI’s time in an ESRC grant application?

27/01/2016 © The University of Sheffield

Methods of collecting data

27/01/2016 © The University of Sheffield

Choose the method that can answer your questions and fits with your event

27/01/2016 © The University of Sheffield

Voting

Comments

27/01/2016 © The University of Sheffield

Any comments on the comments cards?

27/01/2016 © The University of Sheffield

Some comments from FotM

• So much to see! The 83 project is a real eye opener.

• Thermal imaging was fun and carnival. Was an immersive floor

• Great – very inspirational

• Thanks you so much for a wonderful performance. Really enjoyed it. Was great to see the power of music and dance and drama in action and witness the talent in this group! Also great to meet the Professors and staff behind it x

BUT

• I no want it anymore

• Bu bu bu bang!!!!

• My kittens enjoyed the exhibits (I brought him in my bag)

• I like the glass bowl guitar recorder be … is good

• Poor chat

• Peas

• The postit notes

• Carrots?

• The king of the cats

• Phew it smells in this building

27/01/2016 © The University of Sheffield

Social media and media coverage

• Google analytics for website traffic – help available through CiCS

• Storify: to share curated social media stories

• Twitonomy for twitter analytics

• Media reach

27/01/2016 © The University of Sheffield

• Focus groups

• Interviews

27/01/2016 © The University of Sheffield

27/01/2016 © The University of Sheffield

http://www.youtube.com/watch?v=8RlE8hp4WdI

Films and videos

Feedback video from masterclass 1

27/01/2016 © The University of Sheffield

“As a participant, I think it did work and reinforced (in other ways) some of the key issues of that early workshop, so doing the reflection/evaluation in that format was a useful process for me too”.

27/01/2016 © The University of Sheffield

27/01/2016 © The University of Sheffield

Some creative techniques

27/01/2016 © The University of Sheffield

Some creative techniques

Surveys and questionnaires – paper or online

27/01/2016 © The University of Sheffield

Arts & Humanities 9 14%

Medicine, Dentistry & Health

10 15%

Science 11 17%

Social Science 21 32%

Professional Services 8 12%

Other 7 11%

Any comments on the surveys?

27/01/2016 © The University of Sheffield

Observation and photography

27/01/2016 © The University of Sheffield

• Try to record your event with photos

• Observe your audience’s reaction • Count number of visitors • See what kind of engagement is

going on

After your evaluation

27/01/2016 © The University of Sheffield

Data analysis

• How will you analyse the data?

• Anything written on paper takes time

• Online surveys can be good for automated data analysis

• Make sure you think about the analysis at the start

27/01/2016 © The University of Sheffield

Example of output from iPad survey Why did you decide to come the event today?

I thought it would be enjoyable 15

Just passing/ looked interesting 10

I came with a friend 9

I know someone who is involved 8

To hear the speaker 4

To learn something new 3

other 8

Grand Total 57

27/01/2016 © The University of Sheffield

Summary from postcard feedback How much new information did you learn at the event?

A lot of new information 416

A small amount of new information 130

No new information 14

(blank) 19

Grand Total 579

27/01/2016 © The University of Sheffield

Reports • Assistance with planning

• Provides evidence

• Demonstrates value for money

• Reporting to funders

• Learning that can be shared with others and inform future activities

• Influence research

27/01/2016 © The University of Sheffield

Example report • Background to event

• Comments about how it was curated

• Budget

• Audience numbers

• Marketing

• Media coverage

27/01/2016 © The University of Sheffield

Example report • Comment about types of events and which

departments are involved

• Feedback from audience

• Feedback from event leaders

• Comments about what worked and what could be improved

27/01/2016 © The University of Sheffield

Link your findings back to the beginning

27/01/2016 © The University of Sheffield

Example of evaluation

27/01/2016 © The University of Sheffield

Festival of the Mind 2014

• Visitor numbers 26,000

• Visitor comments

• Website stats: Page visit, unique visitors, page views, pages viewed per visit, average visit duration, visitors using a mobile/tablet, non-mobile visitors, new or returning visitors and language settings

• Feedback from academics and partners involved

27/01/2016 © The University of Sheffield

Festival of the Mind evaluation

Visitors

• 579 postcards

• 57 surveys completed using iPads

• Online survey for pre-booked events (335 responses)

• Comment books

• Post-it note comments

• Economic survey for 1 event by external company

27/01/2016 © The University of Sheffield

Festival of the Mind evaluation

Event leaders and partners

• Online survey

• Small amount of interviews and event observation undertaken by an external company

27/01/2016 © The University of Sheffield

Feedback on social media

• Twitter followers: 3127 for main FOTM account with an additional accounts related to projects

• Facebook 1348 likes

• Blogs – over twenty different blogs set up or featured FOTM in the week of the festival.

27/01/2016 © The University of Sheffield

Dr Helen Featherstone, University of Bath

27/01/2016 © The University of Sheffield

Where can I go for help?

• Public Engagement and Impact Team

• Digital Engagement Team – social media

• CiCS – audience response system training, technical help and creative media

27/01/2016 © The University of Sheffield

Guidance

http://www.publicengagement.ac.uk/how/guides/evaluation/resources

http://www.rcuk.ac.uk/Publications/policy/Pages/Evaluation.aspx

http://www.publicengagement.ac.uk/how/guides/evaluation/examples

https://www.sheffield.ac.uk/life - evaluation reports

27/01/2016 © The University of Sheffield

Evaluation

What have you learnt?

How much have you learnt about planning evaluation?

A. A lot of new information

B. A small amount of information

C. No new information

D. Don’t know

27/01/2016 © The University of Sheffield

A lot o

f new

info

rmatio

n

A small a

mount o

f info

r...

No new

info

rmatio

n

Don’t kn

ow

50%

0%0%

50%

How much have you learnt about evaluation tools and when to use them?

A. A lot of new information

B. A small amount of information

C. No new information

D. Don’t know 27/01/2016 © The University of Sheffield

A lot of n

ew inform

ation

A small amoun

t of infor...

No new in

form

ation

Don’t know

42%

0%

8%

50%

How much have you learnt about different kinds of evaluation?

A. A lot of new information

B. A small amount of information

C. No new information

D. Don’t know 27/01/2016 © The University of Sheffield

A lot o

f new

info

rmatio

n

A small a

mount o

f info

r...

No new

info

rmatio

n

Don’t kn

ow

57%

0%

9%

35%

Opportunities for engagement – what is coming up?

• Life festival (April 2016)

• Festival of the Mind (September 2016)

• Off the Shelf (October 2016)

• Festival of Social Science (November 2016)

• Doc/Fest (March and June 2016)

27/01/2016 © The University of Sheffield

For further information contact:

engage@sheffield.ac.uk

@UniShefEngage

#GetEngaged

27/01/2016 © The University of Sheffield

Questions

27/01/2016 © The University of Sheffield

To Discover And Understand.