Creative evaluation tools: Beyond ‘tired and tested’Karen Gray Willis Newson

Agenda

• Aims for the workshop• A very quick introduction to ‘Creative and Credible’

evaluation • Practical exercise: Re-imagining ‘tired and tested’ • Summary

Background: Willis Newson• Specialist arts and health consultancy developing

and delivering projects in acute, mental health and community settings

• Long-term collaboration with Norma Daykin, Professor of Arts and Health at University of West of England

• UWE led research-based evaluations of Willis Newson arts in mental health programmes: Moving On & Arts @Callington Road

• Collaborative research and evaluation work culminated in 2 year Knowledge Transfer Partnership (2010/12) Decorative glazing by Stuart Low in Older

Adult Unit, Callington Road. Commissioned by Willis Newson for Avon & Wiltshire Mental Health NHS Partnership Trust. Photo; Paul Highnam

Evaluation: Three key principles

• Appropriate• Credible• Ethical

All evaluation needs to embody these principles.

Appropriate Appropriate for:

– Evaluation aims and objectives– Participants or stakeholder group– Project context

Some examples:– Focus groups will not give you

quantitative health outcomes data– Validated scales may not be

useful in projects with fewparticipants

– Questionnaires may not be appropriate for people with questionnaire fatigue

Creative exercise – useful for aims setting. Image : Willis Newson.

Credible

• Timing• Objectivity• Robust application and

analysable results

Some examples:– If you tell focus group participants

that the evaluation findings will help secure funding for next year’s project, will they give you the full story?

– How can you measure outcomes for participants if you don’t know where they started?

Image from a ‘Savouring & Flow’ workshop run by Light Box as part of The Happiness Project. Photo: BS6 Photography.

Ethical

• Respect• Confidentiality• Professional integrity

Some examples:– A participant has given consent

for you to use their words, have you told them exactly where those words will appear?

– A participant has agreed to be a case study in your evaluation, but then changes his mind. What do you do? Collage created during a focus group exercise designed to

explore the impacts of a project on participants.

So, why be creative? • To think around problems• To counter evaluation fatigue• To get meaningful responses that

you can’t get in other ways• To make best use of your skills and

those of your participants• To reward and engage your

respondents • To make evaluation more a part of

your project

“Creativity is not an escape from disciplined thinking, it is an escape with disciplined thinking.”

[Jerry Hirshberg]

Reflective practice drawings by Elizabeth Jane Grosse for Arts at Callington Road. Commissioned by Willis Newson for Avon & Wiltshire Mental Health Partnership NHS Trust. Image: Willis Newson.

Tired and tested : Scenario Jenny is the manager of a pilot arts on referral scheme, currently planning how to evaluate it. The project is council-funded and will operate in three GP surgeries in Plymouth, serving the city’s most deprived communities, including those where English is a second language. Intended as a first step intervention for patients with mild to moderate anxiety and depression, it aims to build self-esteem and confidence, develop social networks within the community and offer a positive distraction. Participants will be referred by their GP for a block of 10 sessions. Two local artists will deliver a weekly art session at each surgery covering a variety of different art forms including painting, creative writing and photography. There will be approximately 10 participants in each group, although this number is likely to vary from week to week. A practice nurse will be on hand to offer advice or support or signpost participants to other resources. Jenny needs to evidence the impact the sessions have on participants and advocate for continued funding. She has been asked to use the Patient Health Questionnaire (PHQ9) to measure levels of depression among participants, but she is also eager to understand the subjective impact that the project has on each of the participants and to explore their experience over the 10 weeks.

Tired and Tested: Scenario

Think about:• Who?

– Who will be doing the evaluating?– Who are the participants?

• What?– What form should it take? What will it look like?

• When?– When does it need to happen?

• What are the issues?– Ethics, participant needs, resources, time…

In summary…

• Be appropriate• Be credible• Be ethical• Be creative…

Think about this at every stage:– when you are developing tools– when you are collecting data– when you report and share your

findings

karen@willisnewson.co.uk

Professional development opportunity:

Evaluating Arts, Health & Wellbeing Initiatives

5 days of training Feb – Apr 2014

Further information and training